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ABSTRACT 


The activities and tasks performed by firefighters when responding to emergencies 
caused by asymmetric threats to homeland security mirror the activities and tasks that 
firefighters regularly employ when responding to conventional emergencies. However, 
the learning opportunities created by conventional incidents are not routinely exploited 
for the purposes of preparing firefighters to respond to incidents of asymmetric origin. 
Instead, homeland security training and education is often conducted in a manner that is 
stand-alone and requires a dedicated budget. 

The policy analysis conducted assesses the similarities and differences between 
mitigation procedures and technical skills used when responding to incidents of similar 
nature but different origin and whether or not formal critiques and after action reports 
from conventional incidents can be used to effectively support the long-term sustainment 
of specialized training and education. Efficiency, process values, and robustness and 
improvability are the criterion used to conduct a modified cost-benefit analysis. The 
findings suggest that expanding the scope of formal critiques and after action reports 
from conventional incidents to include “what if’ questions about potential incidents of 
asymmetric origin does facilitate the long-term sustainment of specialized training and 
education programs in a manner that capitalizes on adult and organizational learning 
theory principles. 
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I. 


INTRODUCTION 


A. PROBLEM STATEMENT 

The responsibilities of the fire service related to the asymmetrical threats to 
homeland security fall primarily within the response mission area of the emergency 
management cycle and focus on consequence management.! The asymmetrical threats 
for which the fire service must be prepared to respond include emergencies caused by 
weapons of mass destruction (WMD), such as weapons comprised of chemical, 
biological, radiological, and nuclear materials and/or improvised explosive devices. 
Consistent with traditional fire service priorities, consequence management for WMD 
incidents focuses on life safety, property preservation, and protection of the environment. 

Many of the activities and tasks performed by the fire service during the response 
to emergencies caused by a WMD incident mirror the activities and tasks that firefighters 
routinely employ when responding to emergencies of conventional origin^. However, 
specialized training and education programs that focus on homeland security issues are 
commonly conducted in a stand-alone manner and require a dedicated budget. 
Furthermore, the National Preparedness System promotes the evaluation of capability 
levels for responding to homeland security incidents through staged exercises without 
assessing responder performance during the mitigation of real-world incidents that 
require the use of comparable skill sets. 

Sustainment of specialized training and education programs is often dependent 
upon the availability of grant funding. Since 2002, the fire service is one of many 
disciplines fortunate to receive federal funding dedicated to supporting specialized 
homeland security training and education programs. The costs associated with these 

! Recent efforts to implement the Fire Service Intelligence Enterprise are expanding the scope of fire 
service responsibilities to include prevention activities, but overall prevention activities remain in the law 
enforcement arena and are not addressed in this paper. 

2 The author acknowledges that a situational assessment of emergencies scenes caused by asymmetric 
threats and those with origins unrelated to homeland security will yield some difference among variables 
that must be considered prior to making tactical decisions. Recommendations for incorporating these 
differences into training and education initiatives will be discussed in Chapter VI. 
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programs are allowable homeland security grant expenditures (Federal Emergency 
Management Agency, 2009). While this funding enables many fire departments to 
enhance capabilities, the human resources and time required to conduct these specialized 
programs are often limited and compete with resources required for other fire service 
responsibilities. As fire department corporate budgets continue to become leaner, the 
sustainment of existing specialized training and education programs become increasingly 
contingent upon the receipt of federal grant funding. 

In comparison to the fire service’s historical operational focus, the asymmetric 
threats to homeland security are relatively new. These current day threats are unlikely to 
disappear in the years to come. The influx of funding post- 9/11 created a unique 
opportunity for the fire service to procure specialized equipment and train personnel for 
responding to these new challenges. However, as the novelty of terrorism continues to 
wear off and the threats become more common place, the challenge of preventing 
preparedness and response complacency among first responders increases. Furthermore, 
if future homeland security grant programs are not maintained at the current funding 
level, it is unlikely that specialized training and education programs can be maintained at 
the current levels. 

If specialized training and education programs are not sustained, the fire service 
faces the risk of responders becoming complacent. The consequences of responder 
complacency may include avoidable death and/or injury to first responders and members 
of the public. Additionally, excessive damage to property and the environment may 
occur if proper mitigation strategies are not employed in a timely manner. In order to 
decrease the risk of responder complacency, the fire service must integrate cost-efficient 
strategies for incorporating specialized training and education programs into routine 
operations and training plans. 

B. RESEARCH QUESTIONS 

The two research questions that this thesis seeks to answer are: 

• What are the similarities and differences between capabilities required to 
mitigate emergencies caused by conventional and asymmetric threats to public 
safety? 
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• How can local fire departments use formal critiques and after action reports 
(AARs) to sustain specialized training and education for responding to 
emergencies caused by asymmetric threats? 

C. RESEARCH ARGUMENT 

Many of the activities and tasks identified in the Target Capabilities List (TCL) 
for mitigating emergencies caused by asymmetric threats to homeland security mirror the 
activities and tasks routinely employed by firefighters to mitigate emergencies of 
conventional origin. However, training and education that focus on responding to 
emergencies caused by asymmetric threats are commonly conducted in a stand-alone 
manner and require a dedicated budget. Sustainment of specialized training and 
education is often contingent upon the availability of grant funding. 

Since the year 2002, specialized training and education programs within the fire 
service have largely focused on developing and enhancing the capabilities required to 
mitigate incidents caused by asymmetric threats to homeland security. Fortunately, state 
and local jurisdictions have received federal funding to support these initiatives. 
However, the continuation of federal funding for these programs at the current level is 
not guaranteed, although the asymmetric threats to homeland security that we currently 
face are not likely to be diminished. In order to sustain preparedness capabilities for 
years to come, the fire service must explore possibilities for maximizing training and 
education opportunities that come with a price tag that is cheaper than the costs 
associated with didactic training and full-scale exercises, but which provide similar 
learning opportunities. 

There are a number of similarities between the capabilities required to mitigate 
emergencies caused by conventional threats to public safety and asymmetric threats to 
homeland security. Exploitation of these similarities can help a fire department to sustain 
the capabilities for mitigating homeland security incidents. By incorporating discussions 
on these similarities into formal critiques and AARs from real-world incidents, fire 
departments create a cost-efficient mechanism for increasing the frequency in which 
firefighters actively engage in preparing to mitigate incidents caused by asymmetric 
threats. 
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The infrequent occurrence of homeland security incidents within the United States 
limits the opportunities for firefighters to employ the specialized capabilities developed 
and/or enhanced since 2001 during the response to incidents of asymmetric origin. 
However, urban fire departments routinely employ many of the homeland security target 
capabilities when responding to incidents of conventional origin. For example, 
hazardous material response, decontamination, and search and rescue capabilities are 
used during incidents involving industrial chemical accidents and collapsed or 
structurally unstable buildings. The manner of execution for the core activities and tasks 
required for successful incident mitigation as defined in the TCL remains consistent 
across emergencies that originate from conventional and asymmetric causes. For 
example, establishment of an incident command post in a controlled area outside of the 
hot zone is required for structural fires, hazardous materials incidents, and emergencies 
that result from the detonation of an improvised explosive device. Confirmation that the 
activities and tasks employed to mitigate emergencies caused by conventional and 
asymmetric threats are indeed comparable is required. 

Performance of the activities and tasks included on the TCL during the response 
to incidents that originate from conventional causes is found in standard incident 
documentation completed by members of a fire department. Company officers and fire 
chiefs are required to document actions taken to mitigate an incident by creating a report 
in the federally mandated and standardized National Fire Incident Reporting System 
(NFIRS). A NFIRS report provides an incident summary, including the responding 
companies, mitigation strategy, and sequence of events. Additional incident 
documentation is completed per the requirements of each individual fire department’s 
documentation policies. Chief-level documentation routinely includes specific 
equipment used and tactical procedures employed. At the company officer level, 
documentation commonly includes a detailed accounting of the company’s on-scene 
activities. Types of information in company-specific documentation includes positioning 
of the apparatus, company assignment, sub-assignments of each company member. 
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mitigation activities completed, equipment utilized, victims rescued and treatment 
provided, issues encountered, and information reported to on-scene supervisors and 
command personnel. 

If the activities and tasks employed during the mitigation of incidents caused by 
conventional and asymmetric threats are comparable, then using formal critiques and 
AARs from real-world incidents can help to sustain specialized training and education for 
homeland security preparedness. Integration of “what if’ questions regarding homeland 
security-specific variables during formal critiques of real-world incidents create cost- 
efficient learning opportunities. Integration of such questions provides opportunities for 
firefighters to immediately analyze and apply theoretical information to what was learned 
during the critique discussion. The constructs of adult learning theory and higher-order 
learning support this proposed strategy. This andragogical learning model provides both 
individual and organizational learning opportunities. The incorporation of the higher 
order learning construct of experimentation into the formal critique process supports the 
recommendation of Bazerman (2006) and Henning (2009) to consider multiple causes of 
the problem before deciding upon surprise avoidance tactics. 

Formal critiques, the purpose of which is to critically analyze what worked well 
during the response, what did not work so well, and what can be done to improve future 
responses to similar types of incidents, are not difficult to conduct or document and are a 
recommended best practice within the fire service (Ockershausen, 2008). The conduct of 
formal critiques and development of AARs are not contingent upon the availability of 
specialized funding. 

Formal critiques require the participation of all key personnel involved in the 
incident response. Within the fire service, this means representation from most, if not all, 
responding companies, chief officers, and at times, external agencies. To ensure an 
effective process and optimal participation, department leadership must prioritize 
opportunities for learning and performance improvement. This may require temporarily 
suspending some companies’ availability for assignment to emergency calls received 
during the critique. Critiques that critically evaluate the performance of both the 
company and command levels facilitate experiential learning. 
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Formal critique participants often possess different levels of experience and 
subject matter knowledge. A skilled critique facilitator can take advantage of 
opportunities to educate those in the group who are less knowledgeable on technical 
points that are critical to a successful response, such as ensuring proper strut placement 
during a trench rescue prior to responders entering the space. Although a formal critique 
centers on a historical event, it provides a dynamic opportunity to train for the future. 

Situation-dependant variables prevent two emergencies from being exactly the 
same. However, these variables rarely change the activities and tasks required for 
incident mitigation. Instead, these variables provide different factors to consider when 
devising mitigation strategies. For example, a command post is always established 
during a hazardous materials incident. However, the location of the command post is 
determined based on the chemical(s) of concern and the direction of the wind. 
Firefighters, company officers, and fire chiefs must be skilled at recognizing situation- 
dependant variables and adjusting mitigation strategies accordingly. Formal critiques 
provide opportunities for the participants to consider how hypothetical variables may 
affect the mitigation strategy for the incident being critiqued. 

By introducing “what if’ questions during the critique process, the facilitator 
challenges the group to apply the lessons learned from the analysis of the recent response 
to a potential future situation that is of similar nature but occurred under different 
circumstances. The facilitator is positioned to teach the group about specific variables 
that must be accounted for when responding to incidents caused by asymmetric threats to 
homeland security. The structured but open dialog format of a critique provides a venue 
for group members to experiment with potential decisions by discussing how each option 
or consideration may impact operations. This process facilitates learning from each other 
through the exchange of information and drawing on the collective experience of the 
group without incurring expenses associated with a stand-alone training or preparedness 
exercise. 

The establishment of a balance among the demands of administrative, training, 
emergency response and equipment maintenance responsibilities of a fire department is a 
difficult challenge. The primary responsibility of a fire department is responding to 9-1-1 
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calls in order to protect the health and safety of a jurisdiction’s population. This 
responsibility, coupled with emergency response often being the true passion of those 
who occupy administrative command positions within a fire department, makes it easy 
for the response function to be prioritized at the expense of others. However, if this 
function is not balanced with training and organizational learning, the ability of a 
department to truly learn from their experiences and enhance capabilities is limited. By 
integrating “what if’ questions into formal critiques of real-world incident, department 
leaders achieve both emergency response and homeland security training and education 
objectives. 

The AAR is used to capture the discussion of both the analysis of the incident 
being critiqued and the hypothetical homeland security situation discussed. By 
incorporating the learning points from the critique into the AAR, a fire department 
provides vicarious learning opportunities to firefighters who are not part of the formal 
critique discussion. Written AARs are conducive to wide distribution and incur minimal 
expense for use as a training tool. Therefore, once a fire department commits to the 
formal critique and AAR processes, this training strategy becomes sustainable since the 
financial resources required are minimal when compared to costs associated with current- 
day specialized training and education programs. 

An environment that cultivates learning opportunities and embraces change helps 
an organization to break the perpetual cycle of repeating the same mistakes. If 
department leadership is unwilling to accept constructive criticism and take action to 
address identified areas for improvement, the value of investing time and resources in 
conducting and documenting formal critiques is questionable. Such a department is 
likely to experience predictable surprises in the future. 

Formal critiques that include “what if’ discussions provide a cost-efficient and 
sustainable mechanism for educating first responders on incidents caused by asymmetric 
threats to homeland security. The introduction of new scenarios shortly after critique 
participants have evaluated the strengths and weaknesses of their performance to a 
similar situation provides an opportunity for the group members to solidify the content 
learned through immediate application, a core principle of adult learning theory 
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(Knowles, 1989). A responder’s bank of experiences upon which to draw when 
confronted with a unique situation is expanded by comparing and contrasting the 
similarities and differences of incidents that have different origins. As theorized in the 
Bayesian theory of surprise, “surprise can only exist in the presence of uncertainty” (Itti 
& Baldi, 2005, p. 2). If responders are able to draw upon their experiences to identify 
parallels between responses operations to emergencies caused by conventional and 
asymmetric threats, the probability of surprise decreases and the probability of effectively 
executing mitigation strategies increases. 
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II. LITERATURE REVIEW 


This literature review covers four topic areas that help to answer the two research 
questions posed in this thesis. The first two topics, adult learning theory and 
organizational learning theory, provide insight into the theoretical perspectives that are 
used to guide the development of training and education programs. Information on the 
strengths and weaknesses of the different training and education methods establish 
whether or not formal critiques and AARs are appropriate for consideration as strategies 
to help sustain specialized training and education programs. The third topic, surprise, 
addresses conceptual theories of surprise that relate to first responder preparedness for 
mitigating emergencies caused by asymmetric threats to homeland security. This 
discussion explores the assertion that exploiting commonalities between the capabilities 
required to mitigate emergencies caused by conventional and asymmetric threats helps to 
decrease the probability that firefighters will experience surprise when responding to 
homeland security incidents. The fourth topic, incident critiques, helps to establish a 
baseline understanding of incident critiques and how they are used by public safety 
organizations. This section shows how formal critiques and AARs utilize the principles 
of adult and organizational learning theories and identifies the current gaps in the 
literature in regards to using formal critiques and AARs from real-world incidents as a 
mechanism to sustain homeland security preparedness capabilities. 

A. ADULT LEARNING THEORY 

There are two theories of learning that should be considered when contemplating 
adult education: reinforcement theory and andragogy (Kelly, 2006). 

B.F. Skinner, the behavioral psychologist who discovered operant conditioning 
and articulated the term “reinforcement” as a scientific principle of behavior, claims that 
reinforcements increase the probability that a certain behavior will occur (Epstein & 
Skinner, 1980). In their paper Pavlov and Skinner: Two Lives in Science, Catania and 
Laties (1999) discuss how Skinner built upon Russian psychologist Ivan Pavolv’s Nobel 
Prize winning work on classical conditioning to continue the scientific study of the 


9 



behavior of individuals, which eventually led to the development of reinforcement 
theory. Reinforcement theory asserts that rewards for correct behavior, called positive 
reinforcement, and punishment for inappropriate behavior, negative reinforcement, can 
help an individual to learn (Kelly, 2006). However, in a paper co-authored with his 
graduate student Robert Epstein titled Resurgence of Responding after the Cessation of 
Response-Independent Reinforcement, Epstein and Skinner (1980) question the 
probability of behavior occurrences decreasing once rewards are no longer provided; 
questioning whether or not true learning occurred. Eurthermore, Epstein’s (1991) paper 
Skinner, Creativity, and the Problem of Spontaneous Behavior questions the impetus 
behind the initial behavior that was changed through reinforcement. Without fully 
understanding the behavior and the logic for the behavior, how can the determination be 
made that the behavior change is directly related to the reinforcement? Reinforcement 
theory may be applicable to organizational learning during the early phases of 
transformational learning in order to initiate momentum for change. However, this 
theory should likely not be used in isolation if sustained behavioral change is desired or 
an organization wishes to promote the development of critical analysis and decision¬ 
making skills. 

Alexander Kapp first introduced the concept of andragogy in his 1833 book 
Planton’s Erziehungslehre (Plato’s Educational Ideas) (Peterson, 2009). Andragogy is 
learner focused education that encourages the adult learner to move beyond dependency 
to self-directed learning. Reg Revans, the pioneer of the action learning theory, used the 
principles of andragogy to support the notion that people can use their daily jobs as the 
basis for learning. Revans challenged the conventional model of learning—learn first, 
apply knowledge second. In his seminal work. Revans successfully demonstrated that 
adults who exchange information can use the information learned from each other’s 
experiences to make changes within their own organizations and/or individual practices 
(Margerison, n.d.). 

Malcolm Knowles, considered the founder of adult education theory, used 
andragogy as the foundation for the development of adult education theory. In his theory, 
Knowles encourages the learner to draw upon personal experiences to learn how to 
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perform new tasks and solve problems. Knowles also promotes the immediate 
application of new information learned as a means of solidifying the new concepts and 
behaviors learned. However, in his document Everything You Wanted to Know From 
Malcolm Knowles, Knowles (1989) cautions that learners must not be dumped from the 
pedagogical learning model used primarily for childhood education into an andragorical 
model without orientation on how to adapt to and operate within a self-directed model. 
To omit the orientation phase may cause learning barriers. Knowles argues that by 
nature, adults are self-directed learners and can effectively take responsibility for their 
own learning. 

Revans’ action learning theory and Knowles’ adult learning theory are both based 
on the principles of andragogy and share the same conception that adults can effectively 
learn from their own and other’s experiences. Both Revans and Knowles challenge the 
traditional teacher-centric model of education, instead promoting learner-centric 
education. Charles Margerison (n.d.), one of today’s leading researchers on continuing 
professional development systems, continues to use the work of Kapp, Revans, and 
Knowles by incorporating the principles of andragogy, action learning theory, and adult 
learning theory into workplace learning strategies. 

When combined, reinforcement theory and the principles of andragogy are 
training and education strategies that introduce and help learners comprehend how to 
integrate new information into decision-making processes. Traditional training and 
education programs within the fire service capitalize on both reinforcement and 
andragogy-based theories. Repetitious review of information, techniques, and procedures 
is embraced in the form of training classes and daily drills. The fire services uses 
company schools and tabletop exercises as opportunities to exchange information and 
strengthen abilities for personnel to draw on one’s bank of knowledge and past 
experiences to make effective decisions when confronted with a unique situation. 
Although not explicitly stated in the literature, the formal critique process incorporates 
the principles of reinforcement, action learning, and adult education theories. However, 
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the existing critique process does not routinely provide opportunities for immediate 
application of new information learned—one of tenants of adult education theory 
promoted by Knowles. 

B. ORGANIZATIONAL LEARNING THEORY 

There are multiple frameworks within organizational learning theory that describe 
how organizations learn and the barriers to an organization’s ability to learn. While each 
framework has distinct differences, commonalities are also seen. 

Norwegian researcher Bjarne Espedal (2008) claims that a commonly accepted 
notion found among learning theory frameworks is that lower and higher order learning 
must be balanced for a long-term adaptive or learning process to be successful. Although 
sometimes referred to using different terminology, lower order learning focuses on 
improving existing practices. In the exploitation-exploration framework, Stockholm 
University’s Mikael Holmqvist (2009) defines this concept as exploitation, or focusing on 
specific competencies in order to refine current beliefs and strengthen knowledge in a 
specific area. In the single and double-loop learning framework, Carmeli and Sheaffer 
(2008) describe this concept as the process of error identification and correction, or 
single-loop learning. Smith and Elliott (2007), two leading European researchers in the 
area of crisis management, term this concept first order learning in their work Exploring 
the Barriers to Learning from Crisis: Organizational Learning and Crisis Management. 
Overall, the literature shows agreement that lower order learning aims to improve upon 
the known, enhancing core competencies but not expanding upon existing capabilities. 

In his paper titled In the Pursuit of Understanding How to Balance Lower and 
Higher Order Learning in Organizations, Espedal asserts that higher order learning 
involves using experience to “challenge existing perspectives, routines, and practices and 
to develop new perspectives on the future” (p. 366). This mechanism of learning 
encourages experimentation with the unknown. Holmqvist’s paper Complicating the 
Organization: A New Prescription for the Learning Organization? defines the 
exploitation-exploration perspective as exploration of experiences through creativity, risk 
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taking, and experimentation. From the double-loop perspective, in the paper How 
Learning Leadership and Organizational Learning From Failures Enhance Perceived 
Organizational Capacity to Adapt to the Task Environment, Carmeli and Sheaffer (2008) 
claim that learning occurs when the cause and effect relationship of problems is studied 
so that changes to prevent future occurrences or improvements can be instituted. There is 
agreement within the literature that higher order learning, which involves critical analysis 
of an issue and requires organizational willingness to embrace new and often innovative 
ideas, leads to organizational change. 

Smith and Elliott (2007) identify three perspectives on organizational learning and 
crisis management: learning from crisis, learning for crisis, and learning as crisis. Of the 
three perspectives. Smith and Elliott assert that organizations are most resistant to 
learning from crisis. The availability of information, scape-goating, and lack of 
providing serious attention to the lessons learned from other organizations that have 
experienced a crisis are identified as the most common barriers to organizational learning 
from crisis. Carmeli and Sheaffer’s (2008) study supports the notion that an organization 
can decrease the probability of future crisis occurring if one is able to incorporate lower 
and higher order learning to learn from past experiences, including failures. 

Key factors that contribute to an organization’s ability to learn include 
institutionalization of the learning process, the learning perspective and role of leadership 
in the change process, and the ability to incorporate and balance the different methods of 
learning (Smith & Elliott, 2007; Carmeli & Sheaffer, 2008; Holmqvist, 2009). Eearning 
organizations must overcome the difficulties associated with balancing learning methods 
and guard against the easy temptation of adopting and universally applying a single 
learning method. There is agreement in the literature that the methods of learning 
described in each framework must be balanced in order for organizational learning to 
occur. The right method must be selected in order to achieve the desired outcome. 
Espedal (2008), Holmqvist (2009), and Carmeli and Sheaffer (2008) all agree that if 
methods are not balanced, the organization will inevitably miss valuable learning 
opportunities. 
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Lower order learning builds upon Skinner’s reinforcement theory by promoting 
the continuation of effective behaviors and identifying behaviors to improve or correct. 
From an organizational perspective, higher order learning is similar to action learning and 
adult education theories in that it encourages critical analysis and the use of past 
experiences to identify opportunities for learning and organizational change. Similar to 
adult learning theories, the literature supports combining lower order and higher order 
learning to optimize opportunities for organizational learning. 

The process recommended by the U.S. Fire Administration for conducting formal 
critiques and developing AARs incorporates the use of organizational learning theory 
principles. However, as discussed in the Incident Critiques and After Action Reports 
section of this literature review, the critique process does not currently take full 
advantage of the higher order learning construct of experimentation to develop new 
perspectives on the future. 

C. SURPRISE 

The concept of surprise warrants consideration when discussing capabilities for 
responding to asymmetric threats to homeland security. Webster’s Dictionary defines the 
term “surprise” as “an attack made without warning” (Surprise, 2009). The literature on 
surprise provides much deeper insight on how this phenomenon can be applied to both 
conventional threats to public safety and asymmetric threats to homeland security. 

U.S. Army War College Professor Michael Handel defines the theory of surprise 
in his article The Yom Kippur War and the Inevitability of Surprise as a theory that 
“possesses strong explanatory power” but “forms a weak basis for prediction” (Handel, 
1977, p. 462). One of Handel’s main arguments is that misperceptions on the part of the 
defender facilitate the occurrence of surprise incidents. Incident deconstruction after an 
incident occurrence helps to re-interpret the information known pre-incident with the 
benefit of hindsight. However, this process does not result in accurate predictions which 
may help one to employ surprise avoidance strategies. In his research, Handel (1977) 
argues that predictive abilities are impeded by: 

• The desire to maintain the status quo; 
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• Noise barriers, such as an international background that is asynchronous with 
preparations for attack and the enemy positioning forces to increase the 
difficulty of differentiating between decoy and targets; 

• Overestimation of one’s own capabilities and underestimation of the 
opponent’s capabilities; 

• Inaccurate evaluation of the enemy’s intention; and 

• False alerts that lessen the vigilance against attack. 

In their book Predictable Surprises: The Disasters You Should Have Seen 
Coming, and How to Prevent Them (2004), Max H. Bazerman and Michael D. Watkins 
define predictable surprise as “an event or set of events that take an individual or group 
by surprise, despite prior awareness of all of the information necessary to anticipate the 
events and their consequences” (p. 1). Bazerman and Watkins, two of the leading 
authorities on managerial decision-making, defend the position that predictable surprises 
occur because policy-makers knowingly make decisions that facilitate the incident 
occurrence. These researchers propose six characteristics of predictable surprises: 

• Leaders are aware that a problem exists and that the problem will not fix 
itself; 

• An incident can be expected if the problem continues to worsen over time; 

• Problem mitigation is expensive and the benefits may not be realized for a 
significant amount of time; 

• The benefit of the financial investment made to avert the predictable 
surprise is avoidance of an uncertain, but likely greater cost; 

• The tendency to maintain the status quo often impedes adequate 
preparation for predictable surprises; and 

• Special interest groups that benefit from the status quo typically will work 
to block reform. 

The claim that maintaining the status quo is a factor of surprise is the main point 
of consistency between Handel’s and Bazerman and Watkins’ arguments on why 
decision-makers with all the necessary data fail to correctly anticipate future incidents. 
The authors disagree on the appropriate application of surprise theories and whether or 
not surprise is avoidable. Handel contributes surprise primarily to misinformation and 
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misperception, while Bazerman and Watkins contribute it to the results of conscious 
decisions. The historical climate surrounding the different research endeavors and/or the 
advances in technology that improved access to information between the research time 
periods may contribute to the difference in perspectives. While both used contemporary 
examples at the time of writing to substantiate their assertions, the difference in Handel’s 
examples being traditional military and Bazerman and Watkins’ being asymmetric threats 
to homeland security may also contribute to the diversity among findings. 

Larry Irons, a sociologist who specializes in organizational intelligence, 
organizational learning, and learning communities, uses the concept of predictable 
surprise to further the discussion on organizational learning processes. In his article 
Hurricane Katrina as a Predictable Surprise (2006), Irons claims that incidents with 
vivid outcomes tend to be taken more seriously. Cultivating an environment in which 
personnel remain vigilant to the conditions of predictable surprises becomes an on-going 
management challenge. To engage in surprise avoidance. Irons advocates that managers 
and organizational leaders: 

• Define a clear mission that facilitates workers to function optimally; 

• Promote professionalism; 

• Invest in quality technical training; 

• Cultivate leadership qualities in managers; 

• Maintain an organizational structure that clearly delineates decision¬ 
making responsibilities at each rank; and 

• Value ongoing improvements. 

Harvard Business School Professor Bazerman recently argued in a paper titled 
Climate Change as a Predictable Surprise that focusing on only one possible cause of the 
problem is a common mistake made by leaders in relation to predictable surprise 
avoidance. Ronda Henning, an information assurances scientist, agrees that the failure to 
address multiple possibilities negatively impacts efforts to conduct a comprehensive 
assessment of vulnerabilities and risks, possibly resulting in missed opportunities to 
employ surprise avoidance strategies (Henning, 2009). Use of the elements of a learning 
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organization identified by Irons helps an organization identify opportunities to mitigate 
multiple factors that may contribute to the occurrence of a predictable surprise, increasing 
the likely effectiveness of surprise avoidance tactics. 

Laurent Itti and Pierre Baldi, university-based computer scientists, put forth the 
Bayesian theory of surprise to quantify how the differences between an observer’s 
posterior and prior distributions of beliefs affects their reaction to a stimulus that is 
considered a surprise. Itti and Baldi postulate that only data observations that 
substantially affect the observer’s beliefs yield surprise. The Bayesian theory is based on 
two principles: “surprise can only exist in the presence of uncertainty, which can arise 
from intrinsic stochasticity, missing information, or limited computing resources” and 
“surprise can only be defined in a relative, subjective manner and is related to the 
expectations of the observer” (2005, p. 2). 

The principles of Bayesian theory support Carmeli and Sheaffer’s (2008) 
argument that learning from past experiences decreases the probability of an organization 
experiencing a similar crisis in the future since surprise cannot occur unless a stimulus 
causes a change in beliefs. Learning from previous incidents that share common 
characteristics with the incident at hand can facilitate surprise mitigation. Revan’s action 
learning theory and Knowles adult learning theory support adults using personnel 
experience to learn. 

Many of the principles of surprise found in the literature are commonly used by 
the fire service. However, incorporation of these principles into routine operations by the 
fire service pre-dates the formalization of the theories of surprise. While situation- 
specific variables of an emergency are practically impossible to predict in advance, the 
fire service has successfully identified the types of emergencies and categories of 
potential variables for which they must be prepared to respond. Fire service personnel 
are trained how to mitigate the consequences of incidents caused by these threats, 
decreasing the level of surprise firefighters experience when confronted with an 
emergency. Although the theories of surprise may be relatively new, their tenants are not 
so to the fire service. 
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D. INCIDENT CRITIQUES AND AFTER ACTION REPORTS 

The Department of Homeland Security (DHS) uses the abbreviation “AAR” to 
mean “after action review” (DHS, 2007a). The United States Army uses “AAR” to mean 
“after action report” (Garvin, 2000). Although both terms are related since the discussion 
that occurs during the after action review is typically captured in the AAR, a distinction 
between the two is required since they involve different processes. For purposes of this 
thesis project, “AAR” is used for the “after action report”. The term “formal critique” or 
“critique” is used in place of “after action review.” 

A critique is a training strategy used by many professions and organizations to 
help those involved in an incident or event learn by comparing the expected outcome 
with the actions taken to arrive at the actual outcome (Garvin, 2000; Ockershausen, 2008; 
DHS, 2007a; Turner, 2007; Emergency Response and Crisis Management Technical 
Assistance Center, 2007). Critiques are recommended for both the response to real-world 
incidents and staged preparedness exercises. The findings in David Garvin’s paper U.S. 
Army’s After Action Reviews: Seizing the Chance to Learn agree with the 
recommendations in the U.S. Fire Administration’s Special Report: The After-Action 
Critique: Training Through Lessons Learned (Ockershausen, 2008) on the four main 
questions around which a critique should focus: 

• What did we set out to do? 

• What did we actually do? 

• Of the actions taken, what worked or did not work? Why? 

• What should we do next time? 

The first two questions help discussion participants to develop a common 
understanding of the incident through recreating the timeline. The last two questions 
require the participants to critically analyze the cause and effect relationship of each 
action. The U.S. Fire Administration, an agency within the DHS, supports the position 
that the analytic process helps to identify practices that should be maintained, training 
needs, effectiveness of organizational policies and standard operating procedures, and 
trends or patterns in errors during operations that must be corrected (Ockershausen, 


18 



2008). However, the introduction of theoretical variables that may occur during future 
incidents of similar nature to the incident being critiqued (a.k.a. “what if’ questions) is 
not routinely incorporated into the critique discussion. 

The formal critique methodology of structured but open dialog among participants 
is indicative of an appreciative learning system. Naval Postgraduate School Professor 
Frank Barrett states that appreciative learning cultures “accentuate the successes of the 
past, evoke images of possible futures, and create a spirit of restless, on-going inquiry 
that empowers members to new levels of activity” (Barrett, 1995, p. 4). A critique helps 
an organization to develop the four competencies that Barrett identifies as necessary for 
organization to survival and flourish: affirmative competence- focusing on strengths, 
successes, and potentials; expansive competence- challenging conventional practices; 
generative competence- developing integrative systems that promote realization of the 
effects caused by individual actions; and collaborative competence- facilitating ongoing 
discussion and exchange of ideas. Critiques embody many of Knapp’s principles of 
andragogy and concepts from both Revans’ action learning theory and Knowles’ adult 
learning theory. 

The U.S. Fire Administration, whose mission is to foster a solid foundation in 
prevention, preparedness and response by providing leadership to local fire and 
emergency services agencies, considers the AAR to be the most important part of the 
critique process since it is used to document the lessons learned from an incident 
(Ockershausen, 2008). Although a standardized template for AARs for real-world 
incidents has yet to be created, the U.S. Fire Administration provides an overview of the 
recommended topic areas for inclusion in the written report. The recommendations are 
generic so that they can be applied to a wide range of incident and event types. 

The DHS also advocates the incorporation of critiques and AARs into homeland 
security exercises. The Homeland Security Exercise and Evaluation Program (HSEEP) is 
a capabilities based program that provides national standards for conducting homeland 
security exercises. The Program helps exercise planners integrate elements of the 
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National Preparedness System^ to document achievement of preparedness capabilities in 
a standardized manner (DHS, 2007a). The component of the National Preparedness 
System that most closely tied to HSEEP is the TCE. The TCE provides guidance to state 
and local government agencies on the capabilities that the federal government deems are 
necessary to respond to terrorist attacks and/or natural disasters (DHS, 2007b). The 
HSEEP policy and program documents include a standard template for developing an 
AAR and an improvement plan (IP) and instructions on how to use the documents. 
Through use of the HSEEP AAR and IP, jurisdictions can systematically track progress 
in attaining desired preparedness capabilities. However, given the commonly accepted 
notion of the difficulties involved in simulating real-world conditions during an exercise, 
one may question the extent to which exercise AARs and IPs represent true preparedness 
capabilities since the findings are obtained using simulated conditions. Eiterature on this 
topic is not found. 

Exercise Evaluation Guides, or EEGs, are standardized instruments for assessing 
responder performance during an exercise. Evaluators use EEGs to document player 
performance in relation to each target capability. DHS advocates the use of EEGs 
through the HSEEP program (DHS, 2007a). Christine Bradshaw and Thomas Bartenfeld 
(2009), scientists at the Centers for Disease Control and Prevention who are working to 
develop a national performance measurement system to assess all-hazards preparedness, 
also advocate the use of EEGs to measure performance during exercises. Bradshaw and 
Bartenfeld argue that the strength of the EEC lies in the fact that the evaluation is 
completed by a third party as opposed to self-reported assessments, which likely yields an 
assessment that is comparatively more objective. However, the authors note that EEGs 
commonly emphasize evaluation based on the timeliness of actions. Given that exercises 


^ As a strategy to organize homeland security preparedness activities, DHS established the National 
Preparedness System. The goal of this System is to “achieve and sustain coordinated capabilities to 
prevent, protect against, respond to, and recover from all hazards in a way that balances risk with 
resources” (DHS, 2008). The six components of this System are the 1). National Preparedness Goal; 2). 
National Planning Scenarios; 3). Universal Task List; 4). Target Capabilities List; 5). National Incident 
Management System; and 6). National Response Framework. 
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are based on simulated conditions with artificial timeframes that often are not reflective 
of real-world conditions, one may question the utility of basing an assessment of 
preparedness capabilities on EEG documentation. 

The literature confirms that the fire service has endorsed formal critiques and 
AARs a recommended practice for learning from real-world incidents and preparedness 
exercises. However, no literature is found on integrating the elements of the National 
Preparedness System into critiques and AARs from the response to real-world incidents 
to assess and document preparedness capabilities. Eiterature is not found on possibilities 
for expanding the critique discussion to include theoretical variables for increasing 
opportunities to draw parallels between similar types of emergencies that originate from 
different causes; potentially increasing the number of opportunities for organizational 
learning by fire departments. It is unknown if formal critiques and AARs from 
conventionally-caused incidents can be used by the fire service as a training tool to help 
avoid surprise during incidents caused by asymmetric threats to homeland security. 

E. SUMMARY 

In summary, appreciative learning cultures embrace the principles of andragogy- 
based and higher-order learning theories. Application of the concepts contained within 
each of these theories helps an organization to examine their existing norms in order to 
meet the challenges that they face. Eearning through reinforcement and lower-order 
learning techniques is beneficial. However, strategies based upon these theories alone 
typically focus on only correcting the end result of an problem and do not strive to 
resolve underlying issues, commonly resulting in reoccurrence of the problem. The 
literature argues the importance of organizations learning to embrace multiple learning 
theories and finding a balance among them in order to ensure that learning opportunities 
are not inadvertently missed. 

The principles upon which adult and organizational learning theories are based 
provide support for the U.S. Eire Administration’s recommendation that fire service 
organizations use formal critiques and AARs as tools for learning. These tools 
incorporate concepts of reinforcement, action learning, adult education, lower order 
learning, and higher order learning theories. 
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The literature disagrees on whether or not surprise is avoidable and the 
appropriate application of surprise theory. The contemplation of surprise in relation to 
asymmetric threats to homeland security raises the question of whether or not incidents 
caused by asymmetric threats are truly predictable. 

Finally, conducting formal critiques and developing AARs after real-world 
emergencies and preparedness exercises is recommended by the U.S. Fire Administration 
and the DHS HSEEP program. Critiques and AARs help individuals and organizations to 
learn by critically analyzing the actions taken to identify practices that worked well and 
should be continued and areas that require improvement through additional training, 
purchase of equipment, and/or policy changes. Although standardize templates for 
conducting critiques and writing AARs of real-world incidents do not exist, DHS does 
have templates for exercises and these elements are incorporated into the National 
Preparedness System. However, gaps in the literature exist in regards to the potential for 
using formal critiques and AARs from real-world incidents as tools to help sustain 
specialized training and education designed to prepare firefighters for responding to 
incidents caused by asymmetric threats to homeland security. 


22 



III. METHODOLOGY 


Policy analysis is the methodology selected for analyzing whether or not formal 
critiques and AARs of the responses to real-world incidents can be used to support the 
long-term sustainment of specialized homeland security training and education. The 
subject matter being addressed in this thesis is local implementation of federally- 
prescribed recommendations. The purpose of this analysis is to assess the costs and 
benefits of incorporating a different learning mechanism into the existing training and 
education methods endorsed by the DHS as a strategy to promote the sustainment of 
capabilities required to mitigate emergencies caused by asymmetric threats to homeland 
security. Policy analysis is an appropriate methodology for exploring opportunities to 
improve upon existing practices. 

This policy analysis is based on the claim that firefighters use comparable skill 
sets when responding to certain types of emergencies—such as hazardous materials 
incidents and structure collapses—that may originate from both conventional and 
asymmetric causes. To assess the similarities and differences between the mitigation 
procedures and technical skills used when responding to incidents of similar nature but of 
different origins, an analysis comparing the documentation from real-world incidents and 
the federally prescribed homeland security target capabilities is done. This approach is 
consistent with the National Preparedness System’s HSEEP guidance in which 
preparedness levels are assessed against the federally defined TCE^ . 

A modified cost-benefit analysis is performed to analyze the data from the 
selected cases and comparison sample. Efficiency, process values, and robustness and 
improvability are the criteria selected for the analysis. Each criterion is evaluated 
independently, followed by an assessment of similarities and differences between the 
cases and comparison group. 


4 The capabilities that comprise the TCLs are arranged according to the five categories: common 
capabilities, prevent, protect, respond, and recover. The capabilities that apply to the fire service fall 
primarily within the common capabilities and respond categories. 
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Analysis of efficiency is necessary to assess the financial implications of the 
current and proposed policies. The factors analyzed are human resource and equipment 
expenditures. The results of the analysis are judged in terms of the number of additional 
firefighters who could attend technical training for developing tactical skills required to 
mitigate homeland security incidents if a more cost-efficient strategy for sustaining 
homeland security training and education is adopted. 

Process values analysis (Bardach, 2009, p. 31) is used to assess the opportunities 
available under the current and proposed policies for responders to learn and/or have a 
direct influence on changes to departmental procedures. Each step of the training, 
exercising, and processes for conducting formal critiques is analyzed to identify the 
opportunities for responder participation. The analysis on strengths and weaknesses of 
the current and the proposed policy options is organized in terms of opportunities for 
individual learning, organizational learning, and surprise avoidance. 

An analysis of strengths and weaknesses is also used to assess robustness and 
improvability of the current and proposed policies. The components for this analysis are 
budget, time requirements, administrative complexities, and flexibility for change during 
policy execution. 

A. SAMPLE SELECTION 

There are two issues addressed in this section on sample selection. The first topic 
discussed is the selection of the fire department from which the cases for analysis are 
drawn. The second subject is case selection. 

The Chicago Fire Department (CFD) is the data collection site chosen for this 
thesis project. Chicago is the third largest city in the United States (U.S. Census Bureau, 
2002). However, commonalities in the organizational structure^, demographic 


5 The CFD is run by a Fire Commissioner, with day-to-day operations being managed by a First 
Deputy Fire Commissioner. The First Deputy oversees the five Department bureaus- Operations, Support 
Services, Fire Prevention, Administration, and Labor Relations. The Department operates over 160 fire 
suppression and rescue apparatus, six special operations teams, and 75 ambulances daily. The Department 
responds to over 650,000 calls for service annually (CFD, 2008) 
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characteristics of the population served^, geographic considerations^, and the types of 
incidents to which the fire department responds are found when comparing the 
characteristics of the CFD to those of urban fire departments from cities of smaller size. 
Similar to urban fire departments throughout the country, the CFD complies with the 
codes and standards established by the National Fire Protection Association; creating a 
means for comparing the capabilities among urban fire departments that serve cities with 
different population characteristics and physical attributes. The CFD receives federal 
homeland seeurity grant funding to develop and sustain capabilities for responding to 
homeland security incidents and conforms to all grant requirements for equipment 
purchases, training, and exercises.^ The commonalities found among the characteristics 
of urban fire department suggest that findings from this study, which is based on 
Chicago-specific data, can be generalized and applied to other urban fire departments. 

Four real-world emergencies—two hazardous materials incidents^ and two 
confined space^®, trench rescue^i incidents—that occurred in 2009 are the sample cases 
selected for this policy analysis. A conventional cause triggered each incident and the 


6 Chicago is home to 2.8 million residents who reside in 198 economically, ethnically, linguistically, 
and racially diverse neighborhoods (OEMC, 2006). 

^ The 225 square miles of land that comprise Chicago includes 28 miles of Lake Michigan shoreline, 
over 150 miles of inland waterways, 8 major highways, and 46 movable bridges. Chicago, which is one of 
the busiest recreational harbors in the country, has more than 1335 high-rise buildings, multiple 
manufacturing plants, and numerous entertainment arenas where thousands of people routinely gather. 
There are 193 DHS recognized critical infrastructures/key resources within the City. The City is a national 
hub for rail transportation since it is the geographic location where rail line tracks from the east meet those 
of the western rail lines. (OEMC, 2006) 

^ Per the grant requirements, all equipment purchases must be included as part of the federal 
Authorized Equipment List, all grant funded training must be pre-approved by the DHS, and all exercises 
must conform to the HSEEP guidelines. 

^ The CED classifies hazardous material incidents according to three levels. A Level I hazmat is 
initiated primarily for investigative activities and/or to mitigate incidents involving small quantities of low- 
potential materials. A Level II hazmat is initiated for confirmed incidents involving a moderate to high 
and/or large quantity of hazardous materials, an incident requiring protective clothing above Level D PPL 
(e.g., structural fire fighting clothes), and/or if the incident requires an evacuation to be initiated. A Level 
III hazmat is initiated for extensive incidents that require additional manpower and/or more specialized 
equipment and supplies in order to mitigate, or if an expanded area requires evacuation. (CLD, 2004) 

A confined space is defined as a space that is large enough and so configured that a person can enter 
and perform assigned work, has limited or restricted means for entry or exit and is not designed for 
continuous human occupancy (CLD, 2003). 

1 A trench is defined as a narrow excavation made below the surface of the earth that is deeper than it 
is wide and less than 15 feet wide (CLD, 2003). 
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technical skills employed for mitigation are characteristic of what is commonly required 
for each incident type. Emergencies involving both incident types can originate from 
conventional and asymmetric causes, urban fire departments routinely respond to these 
kinds of incidents, and both are included on the TCL. Furthermore, EEGs for the 
capabilities associated with each incident type have already been developed by DHS. 

Selecting two different incident types for the sample creates the ability to compare 
and contrast findings. The processes for mitigating hazardous materials incidents and 
incidents requiring specialized search and rescue share some common attributes, but 
largely require the use of different technical skill sets. Hazardous material incidents 
typically require the use of metering equipment, modeling software, and specialized 
personal protective equipment (PPE). Incident that require technical search and rescue 
commonly require the use of construction materials (e.g., wood, nails, saws), carpentry 
skills, and at times, heavy machinery. Both require the use of reference materials and 
support from emergency medical service practitioners. Having a sample comprised of 
two incident types should increase the confidence of any recommendations that suggest a 
wider application of findings from the analysis. 

A DHS-approved awareness-level course is the training standard selected for 
comparison with the expanded use of formal critiques and AARs that is being proposed. 
The course. Emergency Response to Terrorism -Basic Concepts, is a standardized 
curriculum that is endorsed by the U.S. Fire Administration that is used to train fire 
service professionals throughout the country. The CFD continues to use this curriculum 
as the baseline terrorism awareness training for all personnel. The expenses for the 
delivery of and participation in this training curriculum are eligible expenditures under 
the federal homeland security grant programs. 

A City of Chicago exercise conducted in December 2009 is the exercise standard 
selected for comparison purposes. This multi-day exercise involved several agencies 
from different disciplines, however, each of the target capabilities and associated 
objectives were exercised in isolation as standalone drills. The CFD exercised target 
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capabilities for WMD/Hazardous Materials Response and Decontamination and Search 
and Rescue (Land-Based). The exercise met the HSEEP requirements and was paid for 
with grant funding from the DHS. 

B. DATA COLLECTION 

1. Case Data 

Documentation sources for each case include Computer Aided Dispatch (CAD) 
reports, electronic NEIRS reports, and AARs. In addition, the author reviewed 
photocopies of company journal entries or performed on-site journal review. 

2. Training and Exercise Data 

Sources of information for the training course include training rosters and 
reimbursement reports. The records include the number of class sessions held, the rank 
of each instructor, and the amount of instructor compensation. Planning records, 
participant rosters, the AAR, and reimbursement reports are the sources of exercise data. 

3. Financial Data 

CED salary and cost-recovery tables serve as the basis for the financial data 
generated. The average salary for all department personnel within a given rank is used to 
determine salary costs. The calculations for personnel ranks covered by collective 
bargaining units reflect contractual obligations, resulting in the use of overtime rates for 
all activities that require re-hire or backfill of union members. Exempt rank personnel 
are not entitled to an increased hourly wage for hours worked in excess of normal 
obligations; therefore, these costs are calculated at the straight time rate. Calculations for 
on-duty activities use the cost-recovery rates. 

An hourly apparatus and equipment charge is routinely included as part of cost 
recovery. Although apparatus and equipment are used during homeland security 
exercises, grant funding is not routinely used to pay for these expenses. Therefore, 
apparatus and equipment costs are excluded from exercise expense calculations. 
Equipment and supply costs for purchases specific to an event are included. 
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C. DATA COMPILATION PROCESS 

1. Exercise Evaluation Guides 

The HSEEP EEG for each target capability is completed using the documentation 
from each case. Each piece of information contained in the documentation is treated as 
an evaluator observation. If observations are recorded in more than one place, they are 
treated as a single observation. Eor observations that reveal inconsistent information, 
such as differences in the amount of time a company spent on-scene as documented in the 
CAD report and company journal, the least subjective source is used. 

To complete each EEG, the author makes an assessment on whether or not each 
task is fully completed, partially completed, not completed, or if the task did not apply. 
Notes explaining how the information contained in the case documentation support the 
assigned measure are inserted under the task description for all tasks ranked as fully or 
partially completed. The Target Capabilities List: A Companion to the National 
Preparedness Guidelines (DHS, 2007b) is referenced as necessary to clarify capability, 
activity, task, and measurement definitions. 

The author completes the EEG Analysis Sheets by identifying the strengths and 
weaknesses of the mitigation activities contained in the documentation. The strengths 
and weaknesses are readily apparent for cases that have a written AAR. The analysis for 
cases without a written AAR is based solely on the interpretation of available 
documentation and the critical tasks associated with each target capability. 

2. Data Tables 

A financial data table is constructed for each case. Using the cost-recovery tables 
and CAD reports, the length of time on-scene for each apparatus, the number of 
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personnel hours per apparatus,^ 21 3 the total cost per apparatus is calculated. The 

numbers in each category for all responding companies are added to produce incident 
totals. 

The subject matter of this thesis focuses on formal critiques and the development 
of AARs as a strategy to sustain specialized homeland security training and education. 
Thus, the cost-recovery data table is used to establish the cost of a formal critique for 
each case. However, it is important to note that the majority of participants in a formal 
critique typically participate while on duty, which means that additional financial costs 
are not incurredi^. Furthermore, critique activities do not include the performance of 
mitigation activities; precluding the eligibility of these activities for cost recovery. 
Although critique costsare not recoverable, the efficiency analysis includes this 
information as a cost comparison when discussing training sustainability. Supervisory 
personnel who were not part of the incident commonly participate in formal critiques, but 
are excluded from cost calculations. Computation of costs is based on each critique 
lasting three hours ^6. 

Although strong similarities may exist, no two incidents or exercises are exactly 
the same. Therefore, personnel and resource requirements will vary between incidents 
and exercises. As part of the efficiency analysis, an estimate of time, personnel, and 


One engineer, three firefighters, and one officer staff an engine company. Four firefighters and one 
officer staff a truck company. Five firefighters and one officer staff a squad company. Two firefighters 
staff a command van. One paramedic and one paramedic officer staff an ambulance. Battalion and 
Paramedic Field Chiefs drive themselves, while Deputy District Chiefs and Assistant Deputy Chief 
Paramedics have drivers. Non-platoon exempt rank personnel drive themselves. Dedicated staff are not 
assigned to support apparatus but are instead assigned to a team company. A member of a team company, 
which is a fully staffed engine or truck company, drives the support apparatus to an incident as required. 
CFD is permitted up to 30 manning variances per day per the union contract. These variances are not taken 
into account for purposes of this study. (Chicago Firefighters Union, Local 2, International Association of 
Firefighters & City of Chicago, 2003). 

^^3 Personnel assigned to support apparatus are accounted for by company of primary assignment. 
Equipment costs only are generated for support apparatus. 

It is the CFD’s practice that companies involved in a formal critique remain in-service on-the-air. 
However, to help facilitate the conduct of the critique, the Fire Alarm Office will prioritize companies that 
are not participating in the critique for dispatch, within defined parameters. 

For purposes of calculating formal critique costs, it is assumed that all participants will be present 
for the duration of the critique. 

CFD critiques typically last two hours. An additional hour is added to reflect time spent traveling to 
and from the critique location. 
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associated expenses for an exercise of similar scope and nature to the incident in each 
case is developed. The estimate comprises the same company configuration and 
personnel numbers dispatched for the case’s response. However, the structure of an 
exercise requires all participants to be present for the duration of the event. During an 
exercise that replicates an actual event, companies that were on-scene for a limited time 
during the real-world incident spend an increased amount of time on-scene in the staging 
area during the exercise. Consequently, the time spent participating in an exercise is 
often not directly proportional to the real world role for some companies. Each exercise 
calculation is based on the actual work time of the incident plus four hours for the 
required pre-and post-exercise activities (e.g., check-in, safety brief, hotwash, site clean¬ 
up). Due to the impracticality of using such a large number of in-service companies for 
an exercise and the fact that reserve apparatus and re-hired personnel are commonly used 
for large-scale exercises, the salary tables instead of the cost-recovery tables are used to 
calculate personnel costs.Expenses for the development of the exercise AAR are not 
included since the AAR is typically written by a salaried employee. However, it should 
be noted that writing an AAR typically takes 12 hours. 

The data table constructed for the homeland security class includes the number of 
sessions conducted, the personnel hours required to deliver the training, and instructor 
expenses. Instructors taught on their day off, which entitles them to the prescribed 
overtime rate for their rank. The table includes expenses related to participation in the 
train-the-trainer course. Participants attended the training while on-duty as part of 
routine in-service training, so no participant expenses were incurred. 

The costs for equipment, supplies, and personnel comprise the homeland security 
exercise data table. The total number of personnel hours encompasses the time spent 
planning, executing, and evaluating the exercise. Data is separated for the hazmat 
incident and technical rescue incident components of the exercise. 


The officer position on a track, engine, or squad may be staffed by either a lieutenant or a captain. 
The officer position on an ambulance may be staffed by either a paramedic-in-charge or an ambulance 
commander. For consistency purposes, all rates are based on the suppression officer being a lieutenant and 
the ambulance officer being a paramedic-in-charge. Both titles are the junior ranking officers. 
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IV. CHICAGO FIRE DEPARTMENT INEORMATION SOURCES 


The two sections in this chapter provide context for the reader on the data sources 
and sample cases used for this thesis project. The first section details CFD processes to 
capture incident response information. The second section provides a brief description of 
each case selected for the sample. 

A. DATA CAPTURE PROCESSES 

The purpose of this section is to validate the reliability of the sources used for 
data collection. With the exception of the AARs, the data capture for each source is 
governed by either a CFD general order or Office of Emergency Management and 
Communications (OEMC) Eire Alarm Officestandard operating procedures. CED uses 
a standardized template for AARs, however, the conduct of formal critiques and 
development of AARs is not governed by a general order. 

Each time a 9-1-1 call is received and a fire company dispatched, the Eire Alarm 
Office creates an incident ticket in the CAD system. The CAD system populates the 
incident ticket with the time, date, location, type of incident, and companies dispatched. 
Information captured for each dispatched company includes the unit number and the 
times of dispatch, departure from the fire house, on-scene arrival, and release from the 
scene. Company officers press a button on the portable data terminal (PDT) that is 
installed in the apparatus to enter departure and arrival times. Since the PDTs are 
connected to the CAD system, all information inputs on the PDT are automatically 
transmitted to the CAD system and entered in the incident ticket. Dispatch personnel 
manually enter remarks in the ticket to document reports provided by company officers 
of on-scene conditions, mitigation activities, resource requests, and status of response 
operations. Incident tickets are archived in the CAD system as printable CAD reports. 


The OEMC’s Fire Alarm Office receives 9-Tl calls and is responsible for dispatch of all fire 
companies. 
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Electronic NFIRS reports are pre-populated with data from the CAD incident 
ticket, but require company and chief officers to manually enter additional narrative 
information. Officers enter mitigation priorities and strategies, the assignment of each 
responding company, the role of on-scene support agencies, and additional resource 
requests and/or notifications into the NFIRS report. Some officers include the tactical 
strategies used by each company to carry out their specific assignment(s). Information on 
injured victims, both those who treated on-scene and released or transported from the 
scene for further medical care, is captured in the NFIRS report. 

For incidents involving hazardous materials, the Hazardous Incident Team officer 
completes an electronic hazmat-specific NFIRS report. This report is appended to the 
main NFIRS report. This report contains information on the type of hazardous materials 
present, circumstances and equipment involved in the release, weather conditions, tools 
used for material identification, and equipment, supplies, and manpower assignments. 
The officer also writes a narrative detailing isolation, identification, notification, 
mitigation, and termination activities. Chicago-specific hazmat tactical worksheets and 
photographs to document the incident are often incorporated as part of the hazmat 
NFIRS. 

Per CFD protocol, each company officer documents all of the company’s 
activities for the shift in the company’s leather-bound journal. For each incident to which 
the company is dispatched, the officer documents the type of incident, the CFD Incident 
Commander, the company’s assignment(s), the specific assignment of each company 
member, tactics employed to complete the assignment, equipment used, and notifications 
made. Situation specific, journal entries may include the activities of other on-scene 
agencies, injuries to responders, equipment that needs to be repaired or replaced, and 
other unique circumstances encountered or actions taken. All companies and chief 
officers maintain journals. 

Chicago Fire Department AARs contain a brief narrative of the incident, actions 
taken for mitigation, and an analysis of what worked well and areas for improvements. 


32 



Some CFD AARs contain additional information such as the impact certain training 
classes have had on operational capabilities and/or CFD’s interactions with external 
response partners. 

B. BRIEF CASE DESCRIPTIONS 

I. Hazardous Materials Incident 

a. Case 1—Level H Hazardous Materials Incident With an EMS 
Plan 

On a July weekday, a 9-1-1 call reporting an explosion at a food 
processing plant is received. The initial responding companies confirm the report and 
identify the source of the explosion as a large outdoor chemical storage tank. A Level I 
Hazardous Materials Incident (hazmat) and EMS Plan I response are initiated; the hazmat 
is subsequently escalated to a Level II. After conducting metering operations to identify 
and establish hot, warm, and cold zones, personnel trained as hazardous material 
technicians don appropriate PPL and enter the hot zone. PH strips and chemical spilfyter 
sticks are used to test run-off and stagnant water for chemical contamination. The 
chemicals involved are identified as glycerine and sulfuric acid. On-scene Fire 
Suppression and Rescue companies assist the Hazardous Incident Team with deploying 
drain covers to prevent chemical run-off from entering the storm sewer and with 
establishing emergency and technical decon. Two plant employees require 
decontamination and emergency medical treatment before being transported to a hospital 
for further medical evaluation and treatment. One injured CFD member requires 
decontamination before receiving on-scene emergency medical treatment and being 
transported to a hospital for further treatment. Representatives from the Chicago 
Departments of Fire, Police, Environment, and Buildings, Common Wealth Edison, 
People’s Gas, and personnel who worked in the plant staff the unified command post. 


An EMS Plan I sends five additional ambulances to the scene, which are above what has already 
been dispatched for the Level II hazmat (CFD, 2007). 
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The Fire Department Incident Commander secures the incident and turns it over to the 
Department of Environment for recovery operations. Incident mitigation took 
approximately four hours. 

b. Case 2—Level II Hazardous Materials Incident 

On a weekday morning in January, a 9-1-1 call reporting a broken 
ammonia pipe in a cold storage facility is received. A Level I hazardous materials 
response is dispatched. The companies who conduct the initial investigation confirm the 
report and escalate the incident to a Level II hazmat. The 17 employees in the facility are 
evacuated. EMS personnel triage and treat the evacuees, all of whom refuse transport to 
a hospital. CLD hazardous materials technicians don Level A PPE^o to enter the 
building and conduct metering operations. The incident is mitigated by shutting off the 
supply valve to the broken ammonia supply pipe. The building interior is ventilated by 
the pre-installed exhaust fans. Lire department personnel conduct metering operations 
outside the building to ensure the safety of surrounding residents during building 
ventilation. One CLD member became ill during the incident for an unrelated medical 
issue and was transported by EMS to the hospital. Incident mitigation took two hours 
and 45 minutes. 

2. Search and Rescue (Land-Based) Incidents 

c. Case 3—Confined Space—Trench Rescue 

On a weekday afternoon in May, a 9-1-1 call requesting emergency 
assistance to free a construction worker trapped in a below grade hole is received. The 
companies initially dispatched to the scene confirm that there one person is trapped in a 
5’ X 5’ X 10’ hole by an earth cave-in. The victim is buried up to his knees in an 
unsupported trench and reports that his legs are pinned underneath a pipe. A full 
confined space, trench rescue response is initiated. Plywood, struts, air bags, and shoring 
are used to stabilize the hole. Shovels and a vacuum truck are used to facilitate the 

Level A PPE is a fully encapsulating protective suit that is vapor and gas seal proof under which a 
self-contained breathing apparatus is worn. 
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removal of the debris trapping the victim. While the victim is still trapped, EMS 
personnel start an IV line on the victim and administer IV fluids, pain medications, and 
other medications as appropriate for a victim with crush injuries. The victim’s blood 
sugar level is monitored throughout the incident. An A-frame is constructed to assist 
with packaging and removing the victim from the hole. Upon removal, the victim is 
treated on-scene and transported to a hospital. External departments brought resources to 
the scene as requested to assist with mitigation operations. Incident mitigation lasted 
approximately five hours and 30 minutes. 

d. Case 4—Confined Space—Trench Rescue 

On a weekend day during August, a 9-1-1 call reporting two victims 
trapped in a below-grade hole is received. The first companies on-scene confirm that two 
victims are trapped in an unsupported trench, with the earth cave-in trapping one victim 
to just above the waist. A full confined space, trench rescue response is initiated. The 
trench is stabilized with plywood, struts, and shoring. Rigging ropes are attached to the 
1st and 2nd floor decks that are above the trench location. Responders use shovels and a 
vacuum truck to remove the soil. One of the victims requires the use of a Harty harness 
and hoist for removal from the trench. EMS monitors the victims throughout the incident 
and upon removal from the trench, transports both to a hospital. Incident mitigation is 
completed in approximately six hours and 15 minutes. 
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V. ANALYSIS AND FINDINGS 


This chapter is comprised of two sections. The first section discusses the 
similarities and differences between the response to real-world incidents and the federally 
defined capabilities that are needed to respond to homeland security incidents. The 
second section details the modified cost-benefit analysis conducted to assess whether or 
not using formal critiques and AARs of real-world incidents is a practical strategy for 
supporting the long-term sustainment of specialized homeland security training and 
education. The modified cost-benefit analysis is divided into three subsections: 
efficiency analysis, process values analysis, and analysis of robustness and improvability. 

A. ANALYSIS OF SIMILARITIES AND DIFFERENCES 

The TCL provides guidance to state and local government agencies on the 
capabilities deemed necessary to effectively respond to homeland security incidents 
(DHS, 2007b). The activities and tasks required to achieve each capability are identified, 
as are the specific actions performed to complete each task. 

EEGs for each capability are structured so that evaluators can assess the 
performance of each activity by evaluating the task individually and then recording an 
overall assessment of progress toward achieving the capability. Achievement of 
activities is measured by assessing the level of task completion and the timeliness with 
which target benchmarks are attained. The categories for task completion are fully 
complete, partially complete, not complete, and not applicable. Benchmarks are 
measured by comparing the targeted time and the amount of time actually required for 
task completion. A narrative is used to record the overall assessment of strengths and 
areas for improvement. 

To assess the similarities and differences between incidents that originate from 
conventional and asymmetric causes, each of the selected capabilities and cases are 
analyzed separately, followed by an assessment of whether or not the case demonstrates 
performance of the capability. This section concludes with a summarization of the 

similarities and differences of the findings for the two capabilities. 

37 



The differences between data collection methods for this study and HSEEP- 
compliant exercises may impact the assignment of level of completion rankings. During 
an exercise, evaluators observe player actions and conversations and make note of their 
observations on the EEG. Exercise evaluators may see or hear things that help to justify 
assigning a greater level of completion than what a company officer routinely documents 
to indicate performance of the same activity. Eor this thesis project, if documentation 
does not exist, it is assumed that the activity or task was not performed. Therefore, some 
of the rankings assigned may be more restrictive than the rankings assigned if an EEG is 
completed by a real-time evaluator. Additionally, exercise evaluation plans permit the 
exercise design team to eliminate some of the TCE activities and tasks from the exercise 
evaluation. This thesis project assesses the level of completion for all activities and tasks 
for each of the selected target capabilities. 

1. Weapons of Mass Destruction/Hazmat Response and 

Decontamination Target Capability 

Eight activities, 46 tasks, and 18 time sensitive benchmarks are identified for the 
Weapons of Mass Destruction/Hazmat Response and Decontamination target capability. 

a. Case 1—Level II Hazardous Materials Incident With an EMS 
Plan I 

Case 1 provides an opportunity to demonstrate performance of all eight 
activities that comprise the WMD/Hazmat Response and Decontamination target 
capability. Case documentation reveals that CED personnel fully completed 35 of the 46 
required tasks, partially completed three tasks, did not complete four tasks, and that four 
tasks are not applicable to the circumstances of this case (see Table 1). CED met the 
target time for 14 of the 17 applicable benchmarks. Although performance of the 
activities for the three benchmarks not met is documented, the lack of recorded 
timeframes precludes an accurate assessment of whether or not the target was achieved. 

Two of the three tasks ranked as partially completed received this 
measurement because the required tasks are not documented in the specified manner. 

The EEG requires the development of a written Incident Action Plan (lAP). Although 
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the case’s plan of action is documented in the CAD report, NFIRS report, and company 
journal entries, this documentation does not meet the written lAP requirement. This 
requirement does not account for the fact that formal lAPs are typically not developed for 
incidents that only last for one operational period since the incident is usually mitigated 
in less time than it takes to develop the lAP. While the conduct of an informal critique 
prior to companies being released from the scene is standard for the CFD, neither the 
occurrence of a critique nor development of an AAR are documented as required in the 
EEG. The incident did not require performance of the third task judged as partially 
complete to the level prescribed in the EEG. Some of the specified actions were 
performed in a scaled manner, demonstrating cognitive ability to correctly identify the 
need and initiate performance execution. 

Eack of documentation is the reason that three of the four tasks identified 
as not completed received this ranking. Eor the fourth task, plume modeling, there is 
neither documentation for activity completion nor an explanation on why plume 
modeling was not required. 

This case demonstrates overall achievement of the target capability 
WMD/Hazmat Response and Decontamination. This case provides an opportunity for 
responders to execute different decontamination techniques, proactive protective 
activities, and use different metering and product identification equipment. 
Documentation of task performance indicates that the response to this real-world 
emergency provided CED personnel with an opportunity to execute the critical activities 
and tasks deemed necessary to mitigate hazardous material incidents that originate from 
asymmetric causes. 
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Table 1. Summary of Target Capability Task Performance 



Total Number of 
Tasks 

Number of Fully 
Completed Tasks 

Number of 
Partially 

Completed Tasks 

Number of Tasks 
Not Completed 

Number of Tasks 
Not Applicable 

Number of 
Benchmarks 

Number of 
Benchmarks 
Achieved 

Number of 
Benchmarks Not 
Applicable 

WMD/Hazmat 
Response and 
Decontamination 
Case 1 

46 

35 

3 

4 

3 

18 

14 

1 

WMD/Hazmat 
Response and 
Decontamination 
Case 2 

46 

25 

6 

10 

5 

18 

14 

4 

Search and 

Rescue (Land 
Based) Case 3 

28 

22 

2 

0 

4 

17 

10 

7 

Search and 

Rescue (Land 
Based) Case 4 

28 

21 

3 

0 

4 

17 

10 

7 


b. Case 2—Level II Hazardous Materials Incident 

Case 2 provides an opportunity to demonstrate performance of all eight 
activities that comprise the WMD/Hazmat Response and Decontamination target 
capability, but the nature of the case limits the number of tasks and actions required for 
incident mitigation. Full completion of 25 of the 46 tasks and partial completion of six 
tasks are contained in the case documentation. Ten of the target capability tasks are not 
completed and five tasks did not apply. All applicable benchmarks are met. 

Again, the primary reason for a task receiving a partial or not complete 
ranking is lack of documentation. Although metering operations are documented and 
metering is typically conducted to establish the hot, warm, and cold zones for a hazardous 
materials incident, documentation regarding the establishment of zones is not found. 
Standard procedures for entry into a hazardous environment that requires the use of Level 
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A personnel protective equipment (PPE) requires establishment of a back-up team and 
emergency decon. While it is probable that both occurred, lack of documentation causes 
these tasks to receive partial and not completed rankings. Development of formal lAPs, 
plume modeling, and the conduct of informal and formal critiques are not documented. 

This case demonstrates achievement of seven of the eight activities listed 
for the WMD/Hazmat Response and Decontamination target capabilities. Insufficient 
documentation exists to determine whether or not decontamination operations, the eighth 
activity, for first responders occurred in accordance with the required tasks prescribed in 
the TCL. This case provides an opportunity for responders to operate in Level A PPE, 
conduct metering operations, and triage and treat victims. Documentation from this case 
reveals that the activities and tasks performed in response to an incident originating from 
a conventional cause are consistent with the activities and tasks that DHS anticipates are 
needed to mitigate incidents with asymmetric origins. 

c. Comparison of Cases 1 and 2 

Both cases provide opportunities for responders to execute the activities 
and tasks that comprise the target capability WMD/Hazmat Response and 
Decontamination. However, neither case alone provides an opportunity to exercise all of 
the tasks associated with each activity. This finding is reflective of HSEEP allowances 
for selecting specific activities and tasks for inclusion in the evaluation that are specific to 
the exercise scenario. 

The level of response required to mitigate each case may be less than what 
is required to mitigate an incident with an asymmetric origin. Eor example, the limited 
number of people requiring decontamination in the cases could potentially be a much 
greater number during a homeland security incident. However, the evaluation of the 
cases suggests that real-world incidents do provide opportunities to perform similar tasks. 
Routine execution increases abilities to successfully perform a task under different 
circumstances that may be more complex in nature. 

In both cases, documentation of certain tasks does not consistently meet 
the standards established in the target capability. Without observing on-scene activities 
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or interviewing personnel involved in the incident response, it is impossible to know 
whether or not the tasks were performed but not documented or if the tasks were just not 
performed. There is consistency between the two cases in the items not documented, 
such as searching the scene for an improvised explosive device, plume modeling, and 
conducting informal critiques. This consistency may be the result of the tasks being 
standard in the response to every hazardous materials incident and therefore not 
documented as a special activity performed. Conversely, the lack of documentation may 
exist purely because the tasks are not performed. Possible reasons for non-performance 
include the lack of requirement by the circumstances of the incident, the tasks not being 
included in departmental operating procedures, and/or neglect by the incident commander 
or company officer. An on-scene evaluator would likely observe if such activities are 
performed but not documented and/or be aware of the reason(s) for non-performance. 
However, the structure of this thesis project limits the extent of observations to existing 
documentation only, impeding the ability to identify the true causes. 

Overall, the cases suggest that the activities and tasks for this target 
capability are routinely performed when responding to incidents of conventional origin. 
The tasks performed are customized to the requirements for incident mitigation. The 
similarities between the responses to conventionally and asymmetrically caused 
hazardous materials incidents create opportunities for using conventionally caused 
incidents to prepare for ones of asymmetric origin. 

2. Search and Rescue (Land-Based) Target Capability 

Seven activities, 28 tasks, 13 time sensitive elements, and four numeric targets are 
identified for the Search and Rescue (Land-Based) target capability. 

a. Case 3—Confined Space—Trench Rescue 

Case 3 provides an opportunity to demonstrate performance of six of the 
seven activities that comprise the Search and Rescue (Land-Based) target capability. 
Case documentation reveals that CFD personnel fully completed 23 of the 28 required 
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tasks, partially completed two tasks, and that four tasks are not applicable. There are no 
missed opportunities to complete a task. Ten of the 17 benchmarks are met, seven do not 
apply. 

One activity in this target capability is devoted completely to searching for 
possible victims. This case’s victim is visualized from incident onset, obviating the need 
to perform search activities. In spite of this, documentation identifies the performance of 
some of the tasks associated with this activity since search and rescue activities are 
systematic, integrated, and commonly share the procedures used for scene control and 
communications. For example, the system used for accountability of search personnel 
and equipment is established since it is the same process used for tracking resources 
allocated to rescue operations. The communication and planning functions that are 
established on-scene for this case and a single visualized victim are consistent with 
procedures used in support of search activities for trapped and missing victims. The 
available documentation points toward four of the six benchmarks for this activity 
identified as not applicable as possibly receiving a fully or partially completed score 
based on the tasks performed. However, since these benchmarks are related to an activity 
that does not apply to this case, they are also categorized as not applicable. 

Two tasks are ranked as partially complete. The first receives this ranking 
since the EEG requires the submission of documentation during the course of the 
incident. Eor operations that only last one operational period, CED policy requires 
documentation to be completed after the incident is secured. Case 3 lasts for one 
operational period. The second task scored as partially complete is sanitation 
precautions. This ranking is assigned since the case does not require a significant number 
of actions to maintain sanitation. Documentation does exist in regards to medical waste 
removal and securing the scene prior to departure, but these activities do not fully address 
the comprehensive definition of sanitation. Partial credit is given since responders 
documented consideration of this task and completion of the limited actions required. 

Benchmarks for this target capability include requesting search and rescue 
assistance from regional and federal partners and extracting victims from different types 


43 



of construction. Resources external to the City are not required for case mitigation. Only 
one type of construction is involved in this incident. 

By and large, this case demonstrates performance of the target capability 
Search and Rescue (Land-Based) for the activities and tasks that apply to the case. 
Although the search activity is not required, this case provides an opportunity to execute 
complex rescue operations. Responders perform rescue operations using equipment 
specifically procured to enhance CFD preparedness for mitigating homeland security 
incidents. Responders demonstrate application of rescue techniques learned while 
attending specialized training that is funded with Homeland Security Grant Program 
funding. Documentation of task performance suggests that this real-world emergency 
provides an opportunity to execute critical activities and tasks deemed necessary by DHS 
to mitigate homeland security incidents that originate from asymmetric causes. 

b. Case 4—Confined Space—Trench Rescue 

The findings from Case 4 are similar to the findings from Case 3. 
Performance of six of the seven activities is documented with 21 of the 28 required tasks 
being fully completed. Case documentation identifies that three tasks are partially 
completed and that four tasks do not apply. Again, there are no missed opportunities to 
perform at least part of an applicable task. Of the 17 benchmarks, 10 are achieved and 17 
do not apply. 

Similar to Case 3, Case 4 does not require search activities and only lasts 
for one operational period. Extensive action to address sanitation and external resources 
is also not necessary. Although issues and accomplishments are documented by 
company officers, a formal critique and AAR are not documented, earning this task a 
score of partially complete. For reasons analogous to the ones listed for Case 3, 
documentation for Case 4 suggests that this case demonstrates achievement of the 
applicable activities and tasks for the target capability Search and Rescue (Land-Based). 
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c. 


Comparison of Cases 3 and 4 


Both cases provide opportunities to execute the activities and tasks that 
comprise the target capability Search and Rescue (Land-Based). Although neither case 
requires search activities, the dual use of support mechanisms for search and rescue 
activities creates an opportunity to demonstrate competence for certain functions related 
to search activities. The search activity and tasks not applicable to these cases are 
important skills that must be mastered to fully achieve this capability. It is likely that a 
different type of conventionally caused incident, such as a building collapse, will provide 
opportunities to perform these tasks. This finding highlights the flexibility of the HSEEP 
for selecting activities and tasks that apply to an exercise scenario, acknowledging that 
not all activities and tasks apply to every scenario. 

The number of victims in each case is small. While the victim number 
does not trigger a response on the scale of a mass casualty incident, the cases provide 
opportunities to utilize the same skills for victim rescue and medical treatment, but with a 
lesser quantity of personnel. Whereas scene control may become more complex for 
larger incidents, additional supervisory personnel are also dispatched for larger incidents 
to maintain an effective span of control. The National Incident Management System 
(NIMS) identifies the standard for span of control as one supervisor for every three to 
seven subordinates (Eederal Emergency Management Agency, 2005). Earger incidents 
may require the establishment of additional levels of supervision, but tactical procedures 
and communication protocols remain unchanged. 

Appropriate use of technical rescue techniques is a commonality found 
between the cases. Specialized training and education is required to learn the appropriate 
rescue techniques but practice is required for skill mastery. Evaluation of case 
documentation in terms of the Search and Rescue (Land-Based) target capability 
identifies that the execution of the technical rescue techniques performed to mitigate 
incidents that originate from conventional causes are the same as the those required to 
mitigate incidents caused by asymmetric threats to homeland security. While specialized 
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education and training remains necessary to develop technical rescue skills, the findings 
from this evaluation suggests that real-world incidents can be used to prepare for a 
possible homeland security incident. 

3. Comparison of Capabilities 

The findings from the cases for both the WMD/Hazardous Material Response and 
Decontamination and Search and Rescue (Land-Based) capabilities imply that similar 
activities and tasks are used to mitigate incidents that originate from both conventional 
and asymmetric causes. Incidents of conventional origin may be of smaller scope than 
those of asymmetric origin, but the mitigation techniques and procedures are largely 
consistent. 

None of the cases provide an opportunity to demonstrate performance of all 
activities and tasks, which reflects HSEEP planning assumptions. Opportunities to fully 
complete some tasks are limited by the size of each incident. The benchmarks are 
consistently met, often being achieved in less than half the amount of the targeted 
timeframe. The geographic confines of the City of Chicago and richness of CED’s 
specialized resources are likely confounding factors for this finding. Eire departments 
that have fewer specialized resources may not be able to fully replicate these findings. 
However, departments that comply with National Eire Protection Agency response time 
targets should be able to initiate response, basic on-scene activities, and the request for 
mutual aid resources required to support specialized operational requirements within the 
targeted benchmarks. 

The cases selected to evaluate the WMD/Hazardous Materials Response and 
Decontamination capability display a greater variability among tasks than the Search and 
Rescue (Land-Based) capability cases. This variance is attributable to the hazardous 
material incidents originating from different causes—a chemical reaction causing an 
explosion vs. a broken pipe leaking ammonia, while both trench rescue cases originated 
from the same cause—earth caving-in on an unsupported trench. This variance 
highlights the importance of not using a single incident or single type of case in isolation 
to assess overall achievement of a capability. Cases 3 and 4 demonstrate that the 
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activities and tasks used to mitigate similar incidents are consistent. Cases 1 and 2 
demonstrate that there can be great variability in the tasks required to mitigate incidents 
with the same classification. 

Documentation, particularly documentation of formal critiques and AARs, is 
consistently a weakness found among cases. Case 3 is the only case for which 
documentation of a formal critique and AAR is found. 

Bazerman and Watkins (2004) assert that a predictable surprise will occur if a 
problem is allowed to worsen over time and/or the tendency to maintain the status quo 
impedes adequate preparation for addressing known deficiencies. Evaluation of the cases 
in terms of the target capabilities shows that the tasks performed during the response to 
conventionally caused incidents are consistent with the tasks required to mitigate 
incidents with an asymmetric origin. Fire departments know what activities and tasks 
they need to perform if called upon to mitigate an incident of asymmetric origin. 
Identifying areas for improvement during incidents of conventional origin and acting 
upon this information to strengthen areas requiring improvement can help a department to 
avoid a predictable surprise. 

B. MODIFIED COST-BENEFIT ANALYSIS 

Three analysis criteria are addressed in this section: efficiency, process values, 
and robustness and improvability. CFD financial data is used to assess efficiency. The 
U.S. Fire Administration’s recommended processes for conducting formal critiques and 
writing AARs, DHS training catalog, and the HSEEP process for planning exercises are 
used to conduct the process values, robustness, and improvability analysis. 

I. Efficiency Analysis 

Efficiency is analyzed in terms of the four cases and the DHS-approved 
awareness level training course and HSEEP-compliant exercise selected as the training 
and exercise examples used to demonstrate the current policy. The results of the analysis 
are judged in terms of the number of additional personnel who could attend technical 
training if a more cost-efficient means of sustaining specialized training and education is 
employed. 


47 



a. Case 1—Level H Hazardous Materials Incident With an EMS 
Plan I 

Incident mitigation took approximately four hours and required 113 
responders, who comprised 46 companies (see Table 2). Responders collectively 
invested 226 hours^i in this incident. The response activities cost $12,642.3622, of which 
100% is eligible for cost recovery from the business owner since the incident resulted 
from a violation of hazardous material handling standards. Although not conducted, a 
formal critique of this incident requires an investment of 339 additional personnel 
hours23, for a total cost of $16,120.68 (see Table 3). An exercise of comparable size, 
scope, and length requires an investment of 904 personnel hours24 and costs 


21 Personnel hours for each case are calculated based on the actual time each fire company was 
engaged in responding to the incident. The number of hours per company is multiplied by the number of 
personnel assigned to the company. The total personnel hours for all companies is added together. 

22 Costs for each case are calculated using the CFD’s cost-recovery salary tables. The number of 
hours per firefighter and paramedic are multiplied by the average salary for the designated rank. 

Equipment costs are calculated based on the length of time each piece of equipment is dedicated to the 
response. The costs for all personnel and equipment are added together. 

23 Time investments in formal critiques are calculated based on the critique lasting two hours, plus an 
additional hour for transit to and from the critique location. It is assumed that all personnel who responded 
to the incident will participate in the critique; therefore, the number of incident responders is multiplied by 
3-hours each. 

24 Personnel hours for exercises are calculated based on the duration of the exercise, plus an additional 
four hours for the pre- and post-exercise activities that are required by HSEEP. 
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$44,858.2425. Additional costs for an exercise include exercise-specific supplies, and 
personnel expenses to plan, control, and evaluate the exercise. 

b. Case 2—Level II Hazardous Materials Incident 

Case 2 lasted two-hours and 45 minutes and required 20 companies staffed 
by 57 personnel to invest a combined total of 68 personnel hours to secure the scene. The 
financial costs for human and equipment resources eligible for cost recovery are 
$3,974.74. A formal critique of this incident requires an investment of 141 personnel 
hours at a total cost of $7,981.08. A comparable exercise costs $16,815.74, requiring a 
317 personnel hour investment. 


25 Exercise costs are calculated based on the average overtime rate for the rank of each participant 
times the number of hours of participation. 
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Table 2. Resource Requirements for Exercises of the Same Scope and Duration as an Incident 



Eength of 
Incident 

Number of 
Responders 

Number of 
Companies 

Number of 
Personnel- 
Hours 

Total Cost of 
Response 

Number of 

Exercise 

Companies 

Total 

Personnel- 
Hours per 
Exercise^ 

Total Cost of 
Exercise 

Case 1 

4 hours 

113 

46 

226 hours 
and 19 
minutes 

$12,642.36 

42 

904 

$44,858.24 

Case 2 

2 hours 45 
minutes 

57 

20 

68 hours and 
23 minutes 

$3,974.74 

17 

317.25 hours 

$16,815.74 

Case 3 

6 hours 
and 30 
minutes 

99 

36 

439 hours 
and 27 
minutes 

$25,004.82 

28 

987 

$52,870.44 

Case 4 

6 hours 
and 20 
minutes 

98 

33 

343 hours 
and 31 
minutes 

$20,297.88 

26 

840 hours 
and 20 
minutes 

$44,560.65 

DHS-approved 
course 

17 months 


77 sessions 

2250 

(instructors 

only) 

$131,007.70 




HSEEP- compliant 

exercise- 

Hazardous 

Materials Incident 

12 hours 

106 




9 companies 
plus role 
players 

926 

$34,528.28 

HSEEP-compliant 
exercise—Search 
and Rescue 

Incident 

12 hours 

35 




16 

546 

$25,570.04 

(personnel); 

$24,000 

(supplies) 


Note. The number of responders is the same for the incident response and the exercise. 

“Four hours are added to the time required for incident mitigation to account for required pre-and post-exercise activities. Personnel-hours include hours for 
planning, evaluation, scene staging, and venue clean-up. 
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Table 3. Resource Requirements for Formal Critiques 



Length 

of 

Critique 

Number of 
Responders 

Number of 
Companies^ 

Number of 
Personnel- 
Hours 

Total Cost 
of Critique 

Total Cost 
of 

Comparable 

Exercise 

Case 1 

3 

113 

42 

339 

$16,120.68 

$44,858.24 

Case 2 

3 

57 

17 

141 

$7,228.86 

$16,815.74 

Case 3 

3 

94 

27 

282 

$15,190.44 

$52,870.44 

Case 4 

3 

82 

26 

246 

$12,271.44 

$44,560.65 


lesser number of companies is listed for attending the critique than the number of 
companies that responded since some support apparatus are staffed by team companies. 


c. Case 3—Confined Space—Trench Rescue 

Mitigations activities for Case 3 lasted six hours and 30 minutes and costs 
$25,004.82. Thirty-six companies with a combined total of 99 responders invested 439 
personnel hours in response activities. Costs for this case are recoverable since the 
construction workers violated Occupational Health and Safety Administration safety 
standards. The formal critique conducted required 282 personnel hours and cost 
$15,190.44. An exercise of comparable size, scope, and length requires 987 personnel 
hours, costing $52,870.44. Additional expenses are incurred for exercise planning, 
evaluation, and control, plus supplies required to properly stage the scene. 

d. Case 4—Confined Space—Trench Rescue 

The confined space- trench rescue for Case 4 required 98 personnel on 33 
companies. The incident lasted six hours and 20 minutes, but required an investment of 
242 personnel hours. Expenses for the response total $20,287.88 and are recoverable. A 
formal critique requires 246 additional personnel hours and $13,176.99 while an exercise 
of similar nature involves 840 personnel hours and costs $44,560. 
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e. DHS-Approved Awareness Level Training Course 

Seventy-eight sessions of the class titled Emergency Response to 
Terrorism: Basic Concepts were conducted over a 17 month time period. Instructors 
invested an average of 28.8 hours per session, with the total instructor expenses for all 
class sessions equaling $131,007.70. Additional expenses are not incurred since students 
participated while on-duty. Although opportunity costs are maximized by conducting 
this training while members are on-duty, delivery of this class took over a year since it 
had to compete with other departmental training mandates. 

/. HSEEP-Compliant Exercise 

The Operations Springboard exercise consisted of a series of drills on 
consecutive days to test isolated target capabilities. The target capabilities exercised 
during this event include the WMD/Hazmat Response and Decontamination and Search 
and Rescue (Land-Based) capabilities. 

Exercise of the target capability WMD/Hazmat Response and 
Decontamination required 926 personnel hours and cost $34,528.28. Expenses can be 
reduced by 640 personnel hours and $21,254 if volunteer role players are used instead of 
candidate firefighters. However, the two week time period between the decision by the 
exercise host to conduct the exercise and the exercise dates precluded the CED from 
recruiting volunteers. Although volunteers themselves may be free, using volunteers 
does require investments in recruitment, training, and on-scene management activities. 
The expense for the 56 personnel hours dedicated by 40-hour employees to exercise 
planning and execution is not included in the financial costs since these positions incur 
neither overtime nor require backfill. 

Approximately $50,000 was expended to exercise the Search and Rescue 
(Land-Based) capability. Expenses include the cost of materials for staging the scene and 
mitigation activities (e.g., nails, plywood, spray paint), personnel expenses for staging the 
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scene26 and player participation, as well as charges for exercise planning and evaluation. 
The exercise required an investment of 546 personnel hours. The 92 hours invested by 
40-hour employees are included in the personnel hour total, but not included in the 
financial expenses since these positions do not incur overtime or require backfill. 

g. Discussion 

The data suggests that conducting formal critiques of real-world incidents 
as a training and education tool is more economical than conducting a staged exercise to 
practice a similar skill set. The average cost of a formal critique for the four cases is 34% 
of the estimated cost of a comparable exercise, with a range between 28 and 43%. 
Formal critiques for the hazardous materials incidents cost an average of 34% of the 
expenses incurred by the HSEEP-compliant exercise, while the search and rescue formal 
critiques cost an average of 28% of the amount of money spent on the exercise. There is 
consistency between the different incident types with respect to the ratio of savings. 

Opportunity costs are optimized by conducting training and education 
while personnel are already on-duty since personnel are paid irrespective of how their on- 
duty time is allocated. Although no additional costs are incurred for on-duty activities, 
assessing the issue in terms of financial impact creates a mechanism for a meaningful 
comparison of options. Eire departments must operate within a defined budget, which is 
often lean. Management is frequently challenged to find ways for accomplishing more 
with fewer resources, necessitating that full advantage be taken of all opportunities for 
cost-efficient learning. 

Dedicating personnel time to participation in formal critiques may 
temporarily make a company unavailable for dispatch. However, if non-participating 
companies can “cover” for the limited time period without operational capabilities being 
negatively impacted, the cost of this learning opportunity has been minimized. As 
demanded by operational requirements, a company can be dispatched from the critique 

The exercise scenario required search and rescue operations in a collapsed structure. To ensure 
safety, trained personnel must stage and secure the venue prior to the initiation of exercise play. Staging 
the exercise venue, which consisted of one collapsed structure staged to permit internal access by players 
and three collapsed structures staged for external play only, required 240 personnel hours, incurring 
personnel expenses of $10,745.76. This cost excludes the 48 hours invested by 40-hour employees. 
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location to the incident scene. While such an interruption may negatively impact the 
quality of the formal critique discussion, it is quite possible that an interruption will not 
occur. On a daily basis, CFD companies spend part of their shift drilling or attending 
mandatory on-duty training (e.g., CPR, NIMS compliance) without interruption. If a 
company is dispatched to an incident during the critique, learning still occurs during the 
time company members participated in the critique discussion. Including formal critiques 
as a routine practice for organizational and individual learning does not change daily 
training requirements or incur additional expenses, but instead minimizes the costs for 
providing an additional opportunity for meaningful learning. 

When firefighters assigned to platoon shifts attend specialized technical 
training classes, another firefighter must be re-hired at an overtime rate to backfill the 
position on the firefighter’s assigned company.27 The special operations training classes 
attended by CFD personnel typically last five days, necessitating that two platoon shifts28 
be covered for each student. The average cost of tuition and backfill for a firefighter to 
attend a hazardous materials technician A or B class is $2,591, with the backfill costs 
increasing for students of greater rank. On average, a class for one of the many technical 
rescue skills costs $3,675 per student of firefighter rank. 

The difference between the estimated exercise expenses and the critique 
costs for the hazardous material incident cases is $38,325, or an average of $19,162.50 
per incident. Comparing the difference in savings between the average cost of a formal 
critique for Cases 1 and 2 and the expenses incurred by conducting the HSEEP-compliant 
hazardous materials incident results in a $22,854 difference. 

Eor the search and rescue cases, the difference between the estimated 
exercise expenses and critique costs is $69,980, or an average of $34,990 per incident. 

27 The Labor Contract between the Chicago Firefighters Union, Local 2 International Association of 
Firefighters and the City of Chicago, Illinois (2003) section 16.4 stipulates the minimum requirements for 
manning. 

28 The CFD uses a three-platoon schedule rotation for all fire suppression and rescue personnel. Each 
platoon shift lasts for 24-hours. Although some firefighters will only require coverage of one platoon shift 
while attending a training that lasts for five consecutive days, the calculations in this paper are based on 
two shifts per individual. This ratio produces a lower number in terms of the number of personnel who can 
attend training for the specified amount of funding available, but ensures that the number of personnel 
identified is the minimum number and that findings are not over estimated. 
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The numbers remain consistent when calculating the number of personnel who can attend 
technical rescue training if the HSEEP-compliant exercise expenses are instead allocated 
for training. 

In summary, conducting formal critiques of real-world incidents that 
require the execution of capabilities listed on the TCE provides a cost-efficient means for 
maximizing learning opportunities. Since response operations and formal critiques are 
conducted while personnel are on-duty and already being paid, no additional financial 
expense is incurred. Eearning objectives are still met, but less expense is incurred. 

2. Process Values Analysis 

Analysis of process values is conducted in terms of opportunities for individual 
learning, organizational learning, and surprise avoidance. The strengths and weaknesses 
of the current policy—DHS-approved awareness level training classes and HSEEP- 
compliant exercises—and the proposed policy option—expanding the scope of formal 
critiques and AARs from real-world emergencies to include “what if’ questions as a 
method for sustaining specialized education and training—are evaluated. The section 
concludes with an assessment of the overall findings. 

a. Individual Learning 

The current policy focuses primarily on a pedagogical learning model. 
Eor both classroom-based trainings and HSEEP-compliant exercises, the designers 
determine the areas of content focus. Although management level staff have input into 
the content focus, frontline firefighters often do not. Eurthermore, the process for 
selection of content is usually not transparent. Both curricula and exercise scenarios are 
designed to address specific learning objectives. While exposing firefighters to certain 
information to facilitate learning is a management responsibility, overly structured 
content may limit learning opportunities if the content is not perceived by participants as 
useful and/or adequate time is not allocated for discussing participant-raised issues and 
interests. 
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To become a successful self-directed learner, Knowles (1989) contends 
that adults must be oriented to the model of andragogy since childhood education focuses 
primarily on the pedagogical model. The formal critique process incorporates the 
recommended orientation. At the critique onset, the facilitator leads the group in 
establishing a common operating picture by recreating the incident timeline (Garvin, 
2000; Ockershausen, 2008). Allowing each company to recount their actions and address 
facilitator-posed questions on what worked and did not work well and the reasons why 
creates opportunities for active engagement by participants. By the time the session 
progresses to the “what if’ questions, participants are established as active instead of 
passive participants in the learning process, completing the transition from the 
pedagogical to the adragogical learning model. 

In spite of the fact that new information may be learned during classroom- 
based training and HSEEP-compliant exercises, the current policy does not guarantee 
opportunities for immediate application of the information, contradicting one of Malcolm 
Knowles’ primary principles of adult learning. Knowles postulates that immediate 
application solidifies information retention. If the time frame between information 
acquisition and the need for application is of extended duration, retention abilities may be 
decreased. This consequence positions learning to be dictated by circumstances that may 
be beyond one’s control instead of empowering the learning to take charge of their 
learning environment. 

The traditional formal critique discussion focuses on what went well, areas 
for improvement, and changes that should be made to operations for a future response of 
similar nature (Garvin, 2000; Ockershausen, 2008). Expanding the critique discussion to 
include “what if’ questions for similar incidents with an asymmetric origin provides 
opportunities for immediate application of the new information learned. Knowles (1989) 
asserts that immediate application of new information helps to solidify the concepts and 
behaviors learned. By influencing the direction of the discussion and actively 
participating, participants take responsibility for their own learning while increasing their 
bank of experience upon which to draw when faced with a challenging situation. 
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Delivery of the DHS-approved awareness level training course to all 
Department members required 78 sessions conducted over a 17-month period of time. 
The structured curriculum requires uniformity of delivery by the instructors. The time 
constraints for delivery limit the opportunities for in-depth discussion on areas of interest 
to class participants, making the class more instructor-focused as opposed to learner- 
focused as recommended by Kapp (Peterson, 2009). The effect of the repetitive 
curriculum delivery by the small group of instructors is unknown. Instructors possibly 
became more knowledgeable about the content areas during the 17-month program, 
which can improve the quality of the program delivery. Conversely, it is also possible 
that the repetition may cause instructor enthusiasm to wane, leading to less effective 
delivery of the material and/or possible dilution of content. 

Action learning theory asserts that adults can use their daily jobs to 
exchange information with peers as a means of learning from collective experiences 
(Margerison, n.d.). The proposed policy capitalizes on the use of formal critique 
participants’ recent experiences with responding to a conventionally caused incident to 
frame the discussion on responding to a similar incident that originates from an 
asymmetric cause. Participants are provided with opportunities to actively direct the 
learning process by exchanging information from previous experiences to resolve 
proposed challenges. Although firefighters routinely respond to similar types of 
incidents, the unique circumstances of each response—in terms of conditions of the 
incident venue, the unique make-up of personnel staffing the responding companies, and 
individual company assignments—makes the experiences of each firefighter unique. It is 
conceivable that two firefighters of equal training and assignment will learn different 
lessons from the response to the same incident. Providing a structured but open forum to 
exchange information on application of activities and tasks to mitigate an incident 
focuses the discussion on the key areas of importance to the learner while creating 
vicarious learning opportunities for other participants. 

Instructor-led training programs require direct participation by all 
individuals who wish to learn the content covered. Although training materials may be 
made available to non-attendees, the opportunities for vicarious learning are limited. The 

57 



HSEEP format for exercise AARs is neither user-friendly nor conducive to use as a 
training tool. The current policy does create learning opportunities for actual 
participants, but opportunities for vicarious learning are extremely limited. 

The AAR developed from the formal critique discussion creates a 
vicarious learning opportunity for firefighters who were not part of the incident response 
and/or the critique. This document can be made widely available as content for company 
schools, independent study, and reference material. Although the circumstances of the 
incident are unique, those using the AAR for vicarious learning can still relate to the 
content since there is a high likelihood that the reader has been involved in a response to 
a similar type of incident. This association helps to create a vivid image when one 
chooses to use an AAR as a tool for self-directed learning. Research by Irons (2006) 
demonstrates that incidents with vivid outcomes tend to be more memorable, increasing 
the effectiveness of this learning strategy. 

b. Organizational Learning 

The current policy focuses primarily on improving existing practices, 
which is commonly referred to as lower order learning (Espedal, 2008). Instructor-led 
trainings and HSEEP-compliant exercises focus on developing and strengthening 
specific, pre-determined capabilities. These training mechanisms are important tools for 
promoting the development of core competencies required to perform desired 
capabilities. However, these training methods commonly limit the opportunities for 
learning to a fixed set of objectives (Holmqvist, 2009). Eor example, instructor-led 
trainings are based on a preset curriculum and HSEEP-compliant exercises commonly 
have artificial timelines and scenarios that are designed to test specific aspects of existing 
policies. Additionally, exercise critiques involving all of the participants are commonly 
conducted immediately following the conclusion of the exercise; limiting the amount of 
time participants can spend critically analyzing events and possible solutions for areas of 
identified deficiency. While the existing policy provides opportunities for organizations 
to identify and correct areas of weakness, forms of lower order learning do not promote 
the creation of new capabilities (Smith and Elliott, 2007). 
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The proposed policy of expanding the scope of formal critiques and AARs 
to include “what if’ questions provides opportunities for using real-world experiences to 
critically evaluate existing practices, experiment with making critical decisions, and 
discuss potential policy changes that may be necessary to improve upon existing 
capabilities. This learning methodology is consistent with the principles of higher order 
learning as described by Espedal (2008), Holmqvist (2009), and Carmeli and Sheaffer 
(2008). Learning organizations must critically analyze issues and develop innovative 
solutions. The proposed policy provides a means for front-line and management 
personnel to work together in a constructive manner to study the cause and effect 
relationship of actions. The intent of this policy is to promote learning from experiences 
and to memorialize effective practices, while recommending ways to improve upon areas 
of defined weakness to expand existing capabilities. 

The training strategies embraced by the current policy focus singularly on 
learning for crisis (Smith & Elliott, 2007). Learning for crisis is a fundamental principle 
of fire service preparedness initiatives for a jurisdiction’s specific hazards. However, 
Carmeli and Sheaffer (2008) have shown that learning/rom crisis, not learning/or crisis, 
is a key factor in decreasing the probability of future crisis occurrence. These researchers 
recommend that strategies for both learning for and from crisis be incorporated into 
training strategies aimed to provide opportunities for organizational learning. Learning 
for crisis provides little opportunity for frontline personnel to contribute to overall 
organizational learning. 

Conversely, the proposed policy is designed to help all levels of 
organizational responsibility to learn from crisis, the learning strategy that organizations 
are most resistant to using (Smith & Elliott, 2007). The process used to conduct formal 
critiques addresses the most common barriers for organizational learning from crisis: 
availability of information, scape-goating, and lack of attention to lessons learned, while 
providing opportunities for contribution by all stakeholders. The formal critique process 
embraces the constructs for developing an appreciative learning culture, which has been 
shown to support organizational learning (Barrett, 1995). 
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An organization’s ability to learn is commonly affected by its capacity for 
institutionalizing the learning process and balancing the different methods of learning 
(Smith & Elliott, 2007; Carmeli & Sheaffer, 2008; Holmqvist, 2009). Formal critiques 
and AARs cannot be used as an isolated organizational learning strategy, but must instead 
be used in combination with lower order methods such as instructor-led training and 
staged exercises. The minimal to no additional expenses incurred when conducting 
formal critiques and writing AARs removes the financial challenges commonly 
encountered when expanding training programs. The level of commitment by 
organizational leadership to learn from crisis by institutionalizing the proposed policy is a 
primary consideration for whether or not this strategy can be successfully incorporated 
into existing homeland security training and education programs. 

c. Surprise Avoidance 

Surprise avoidance is one of the purposes for homeland security training 
and education initiatives. Although the current policy does help to prevent the 
occurrence of surprise by increasing one’s knowledge in a specified area, the policy’s 
effectiveness is limited because of the minimal stakeholder involvement in accounting for 
two factors known to contribute to surprise. Both noise barriers and the overestimation 
of capabilities can create preparedness vulnerabilities when the current policy is used in 
isolation (Handel, 1977). 

Noise barriers may cause an organization to misallocate preparedness 
resources. Noise can be in forms such as grant requirements that do not match a 
jurisdiction’s primary risks, competing organizational responsibilities, or the inability to 
effectively process the volumes of threat-related information available to accurately 
assess risks. The current policy for classroom-based training and education limits 
preparedness funding expenditures to course curricula that have been pre-approved by 
DHS, largely excluding stakeholder participation in content selection. Although the 
DHS-approved training catalog is extensive, and there are mechanisms in place to request 
the approval of additional curricula, the lengthiness of the approval process and time 
requirements for implementation make selection of training programs already approved 
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by DHS a more time efficient decision. If the topics that comprise a jurisdiction’s 
primary risks are not included in the catalog, budget constraints and competing priorities 
likely prevent a fire department from pursuing the new training, decreasing the possibility 
that input from frontline personnel will influence training decisions. As discussed by 
Bazerman and Watkins (2004), the tendency to maintain the status quo often negatively 
affects abilities for effectively preparing to confront predictable surprises. Noise barriers 
may influence the ability of a fire department to move beyond existing levels of 
preparedness. 

The proposed policy option creates a means for stakeholder involvement 
in the decision to memorialize existing policy and make recommendations for policy 
improvements. The collaborative efforts to critique the incident addresses the factors 
identified in the Bayesian theory (Itti & Baldi, 2005) as contributors to surprise. 
Collaborative discussion on how to apply the lessons learned from a real incident to a 
similar incident with a different origin helps to decrease the uncertainty of expectations 
because the cause and effect relationship of actions is studied. Confidence in 
expectations of actions by all stakeholders contributes to surprise avoidance at both the 
individual and organizational levels. 

Successful execution during an exercise often equates to perceptions of 
capability mastery, which may or may not be accurate when attempting to execute the 
same skills under different conditions. The nature of exercises makes true simulation of 
real-world conditions difficult. Practicing tactical procedures, communications, and 
decision-making skills is beneficial; however, it is questionable as to whether or not 
exercises truly provide opportunities for learning how to manage and/or avoid surprise. 
Furthermore, the opportunities for firefighters to influence the policy process are limited. 

In staged exercises, planners routinely develop the scenario to test existing 
capabilities and ensure that allocated resources for the event match the requirements for 
successfully meeting exercise objectives. The structure of an exercise usually limits the 
cause of the problem to the pre-defined scenario parameters, limiting opportunities to 
practice comprehensive assessment skills required to employ surprise avoidance 

strategies in real-world situations (Henning, 2009). Injects may be incorporated into an 
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exercise to evaluate abilities to modify operations when additional information is 
received. However, the scenario and injects are commonly developed to assess a limited 
number of activities and tasks and may not reflect realistic conditions. Exercise 
participants may be afforded learning opportunities, but the breadth of opportunity may 
be limited to the pre-defined scope of learning selected by the planning team. As Itti and 
Baldi postulate, “surprise is related to the expectations of the observer” (2005, p. 2). In 
staged exercises, participants are commonly briefed on the scenario in advance of the 
event and have some forewarning of what to expect, limiting the ability to practice skills 
for dealing with surprise. 

The proposed policy creates opportunities for firefighters to actively 
engage in helping the organization to avoid surprise and learn how to effectively manage 
situations in which surprise is encountered through the creation of an appreciative 
learning culture (Barrett, 1995). By expanding the scope of formal critiques and AARs to 
include “what if’ questions, frontline personnel and administration are positioned to learn 
collaboratively from past experiences in order to prepare for future situations in which 
similar challenges may occur under a different set of circumstances. Unlike a staged 
exercise in which the scenario is developed by a small group, the scope of the critique 
discussion is dictated by the participants as they work together to recreate the incident 
timeline and identify strengths and weaknesses of the decisions made and tactical 
procedures employed. As recommended by Henning (2009) and Irons (2006), the 
critique process creates opportunities to identify the multiple factors that may contribute 
to surprise and the tactics that can be used to avoid surprise while optimizing 
participation by the different levels organizational hierarchy stakeholders. 

The proposed policy option also capitalizes on the principles 
recommended by Irons (2006) to avoid surprise. Stakeholder involvement in identifying 
and problem-solving areas for improvement promotes a common understanding of the 
overall mission while cultivating leadership skills in those of lower organizational rank. 
Engagement by frontline personnel in discussions about training needs helps management 
to allocate resources to training initiatives that most advantageously create meaningful 
learning opportunities for the end user. 
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d. Summary 


The analysis of process values suggests that both policy options 
successfully create learning opportunities. The current policy primarily provides learning 
opportunities that focus on content that is pre-determined by a select group of individuals. 
Conversely, the proposed policy option creates a mechanism for the actual learners to 
influence both the content to be learned and organizational policy. Therefore, the 
proposed policy is found to be a more advantageous option when process values are a 
priority consideration in the selection of strategies to create an optimal learning 
environment. 


3. Robustness and Improvability Analysis 

Robustness and improvability of a policy are directly related to field 
implementation (Bardach, 2009). The analysis of this criterion is done as a comparison 
between the current and proposed policy options for each of the selected constructs - 
budget, time requirements, administrative complexities, and flexibility for change during 
policy execution. 


a. Budget 

The current policy requires a dedicated budget for execution of each 
training and education event. If federal funding is used to support an initiative, all 
expenditures must comply with federal requirements. Using the data from the DHS- 
approved awareness level course and the two HSEEP-compliant exercises used as case 
comparisons, the average cost per personnel-hour invested in the execution of activities 
in support of the current policy is $52.42. Reducing this cost may be possible if a greater 
number of on-duty personnel are used as exercise participants and instructors are detailed 
from their daily assignments to teach. However, such strategies shift instead of reduce 
the burden. On-duty, exercise participation causes a large number of companies to be 
unavailable for dispatch assignment for an extended period of time, possibly creating 
operational vulnerabilities. Although detailing personnel to instructor positions does not 


63 



negatively impact the training budget, the overtime budget for operations increases since 
firefighters must be re-hired to fill vacancies created by detailed personnel in order to 
maintain manning requirements. 

Alternatively, execution of the proposed policy does not incur expenses 
that are in addition to the existing financial obligations required to support daily 
operations. Formal critiques are conducted and AARs developed as part of routine 
operations for on-duty personnel. Duplication of AARs for wide distribution does incur 
expense, but the cost is negligible. Furthermore, use of technology to electronically 
distribute and archive documents reduces duplication costs. Although there are costs 
associated with establishing and maintaining technological systems, these systems are not 
exclusively required for execution of the proposed policy and should therefore not be 
attributed as such. 

b. Time Requirements for Implementation 

The HSEEP doctrine subscribes to a standardized template for planning 
that includes a series of planning and coordination meetings (DHS, 2007a). A minimum 
of three meetings to plan an exercise and one post-exercise debriefing meeting are 
required. HSEEP recommends that these meetings occur over a period of many months. 
Additionally, an investment of time is required between meetings to coordinate individual 
agency operations, secure the venue, and acquire resources for exercise support (e.g., 
port-a-potties, props, consumables). Time is also required to develop HSEEP documents 
(e.g., situation manual, master event scenario list, EEGs, safety briefing, role player 
cards) and to stage the venue. The size of the exercise impacts the number of individuals 
involved in planning and execution activities, which directly impacts the total number of 
hours invested. 

Execution of HSEEP-compliant exercises requires that participants - 
players, role-players, evaluators, controllers, and observers—dedicate time in excess of 
the hours spent on exercise play. Participants invest time in picking up equipment before 
traveling to the venue, completing designated check-in procedures, attending a pre-event 
safety brief, participating in a hotwash immediately upon exercise conclusion, and 
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cleaning up the exercise venue and returning equipment and supplies to the appropriate 
locations. Data from the HSEEP-compliant exercises included in this study suggests that 
four hours per participant for pre and post exercise activities is a reasonable estimate of 
additional time requirements. Eurthermore, some participants may have limited roles 
during the exercise but be required to be present for the event duration, creating 
inefficiencies in time usage. 

In addition to hours spent on direct delivery of a DHS-approved training 
curriculum, investments of time are required for attending train-the-trainer sessions and 
coordinating location and participant logistics. An average of two hours per instructor 
per session in excess of instruction time were required to effectively execute the DHS 
course included in this thesis project. 

The U.S. Eire Administration recommends that formal critiques occur 
within the days immediately following the occurrence of the incident to be critiqued 
(Ockershausen, 2008), which limits the time between the decision to conduct a critique 
and actual execution. While an investment of time is required to prepare for facilitating 
the critique discussion, this investment is commonly limited to a few hours by a few 
individuals since re-creating the event timeline and scenario and conducting analysis is 
accomplished by the participants during the critique discussion. Producing the AAR does 
take time, but again this investment is typically limited to a few individuals. 

c. Administrative Complexities 

Execution of both the current and proposed policies involves 
administrative complexities. But the complexities of each are different. 

To maintain operational capabilities, HSEEP-compliant exercises 
commonly require that personnel be rehired as exercise participants. Procedures for re¬ 
hiring CED union employees are governed by labor contract rules, which treat individuals 
of identical rank the same, irrespective of specialty training level. Homeland security 
exercises frequently require participation by firefighters with some level of advanced 
knowledge and training. This requirement necessitates that management confer with 
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Union officials to negotiate special re-hire arrangements in order to avoid the financial 
expense of personnel grievances for non-compliance with union contract rules. 
Negotiation requires a time investment and possibly concessions on a different issue in 
order to optimize the benefit gained from exercise participation. Grant funds cannot be 
used to pay grievances related to an exercise that are found in favor of the plaintiff even 
if a grant was used as the funding source for the exercise; requiring that corporate funds 
be allocated for this additional expense. 

Personnel assignment to attend formal critiques also requires scheduling 
coordination to ensure that operational capabilities are maintained. However, the 
procedures to accomplish this are no different than those executed for assigning 
companies to an incident and changing the location of non-assigned companies to ensure 
coverage throughout the jurisdiction. Dispatch personnel frequently execute these 
procedures, which do not require advanced planning and negotiations since standing 
protocols already exist to facilitate dispatcher decision-making. 

HSEEP recommends that the exercise planning process be completed over 
a number of months (DHS, 2007a), with 6-12 months being a realistic timeframe. 
Although this recommendation provides the time necessary to fully develop the 
objectives, scenario, and coordinate logistics with all stakeholder agencies, the extended 
timeframe also increases the probability that changes in organizational priorities and 
exercise design team members will occur. Changing priorities may lessen the resources 
that are available for dedication to an exercise or make the need to conduct the exercise 
obsolete. Personnel changes increase time investments since new members must be 
briefed on historical planning and decisions, as well as possibly re-negotiating the 
exercise objectives and scenario to incorporate the new member’s input. 

Duplication and distribution of AARs follows the existing process for 
distribution of internal paperwork. Hardcopies of AARs can reach all fire houses on the 
same day they are duplicated. Use of technology to distribute AARs electronically 
expedites the process. 
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d. Flexibility 


The proposed policy is more flexible than the current policy in terms of 
implementation and meeting the learning needs of those affected. HSEEP-compliant 
exercises and delivery of DHS-approved curricula are frequently dependent on the 
availability of specific resources—dates, venue, stakeholders, and resources—and these 
events are routinely planned to accommodate a negotiated scheduled. If an event must be 
cancelled, the process for coordinating all elements to reschedule within the desired 
timeframe can be a very complex process and often involves re-arrangement of other 
commitments. Alternately, formal critiques are scheduled in close proximity to the date 
of the incident being critiqued, facilitating the ability of critique planners to select a time 
and date that is sensitive to current operational schedules and conducive to accomplishing 
the critique goals and objectives. If an emergency necessitates that the critique be 
postponed, it is likely that the group can reconvene later in the day and/or be re-scheduled 
for the subsequent workday since department administration can make decisions on 
resource allocation. If companies are needed for dispatch during the critique, the rest of 
the participants can still continue during their absence. The focus of the discussion may 
change, but learning will still occur. 

Delivery of DHS-approved curricula requires instructors to cover the pre¬ 
approved content and provides little flexibility for the instructors to change the course 
content. Often times, only minimal time for questions or for deeper exploration of areas 
of special interest to participants is incorporated. Conversely, the format of structured 
but open dialog recommended for formal critiques provides opportunities for greater 
discussion on areas of particular concern to participants (Ockershausen, 2008), embracing 
the concept of adult learning theory supported by Kapp (Peterson, 2009), Revans 
(Margerison, n.d.), Knowles (1989), and Margerison (2005). Additionally, if the 
facilitator determines that the critique is not progressing in a constructive manner or that 
learning is not being optimized, the critique format allows for the facilitator to change the 
direction of the conversation to topic areas that may better promote participant learning. 
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e. Summary 


The findings from this analysis suggest that the proposed policy is more 
robust and provides greater opportunity for improving upon the process mid-course than 
the current policy. Formal critiques and AARs create minimal financial burden, require a 
limited amount of time for pre- and post-event activities, and are flexible to accommodate 
both learner needs and organizational responsibilities in a simple and straightforward 
manner. HSEEP-compliant exercises and DHS-approved training courses do provide 
learning opportunities, but also require execution in a manner that is neither as robust nor 
flexible in implementation as formal critiques and AARs. 

C. CHAPTER SUMMARY AND CONCLUSIONS 

In Chapter I, an analysis of the similarities and differences between the response 
to real-world incidents and the federally defined capabilities that are needed to respond to 
homeland security incidents is conducted. The four cases are examined to assess whether 
or not the targeted activities and tasks are performed and the benchmarks met in a manner 
that is consistent with prescribed standards. The findings from this analysis reveal that 
the real-world incidents do provide opportunities for executing the same tactics and 
procedures that DHS predicts are needed to mitigate a homeland security incident, 
although on a smaller scale. 

Consistent with the HSEEP doctrine, none of the incidents in isolation provided 
an opportunity to perform all activities and tasks associated with a specific capability. 
The variability provided by the different case types to execute different activities and 
tasks suggests that the proposed policy should integrate the use of multiple incident types 
and causes of origin within a single incident type category in order to optimize learning. 

Documentation of a formal critique and AAR exists for only one of the four cases, 
suggesting that opportunities for learning from crisis are not consistently being used 
advantageously. The process of learning from crisis can help both individuals and 
organizations identify areas of deficiency and initiate corrective actions, decreasing the 
possibility of a predictable surprise occurring. 
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In summary, the analysis in Chapter I found that there are many similarities 
between the response activities for similar type incidents that originate from both 
conventional and asymmetric causes. The exploitation of learning from conventionally 
caused incidents can create opportunities for preparing to respond to incidents with an 
asymmetric origin that may be of greater size. 

A modified cost-benefit analysis is conducted in Chapter II. The three criteria 
used in the analysis are efficiency, process values, and robustness and improvability. 
Efficiency is analyzed using CFD financial data for the four cases and examples of 
current policy—DHS-approved awareness level training and HSEEP-compliant 
exercises. Process values and robustness and improvability are analyzed in terms of the 
federal recommendations for conducting formal critiques, DHS-approved training 
classes, and HSEEP-compliant exercises. 

The findings from the efficiency analysis put forth that conducting formal 
critiques of real world incidents is a more economical method of training and education 
than conducting staged exercises that are designed to practice similar capabilities. Since 
both incident response and formal critique activities are conducted while firefighters are 
on-duty and already being paid, opportunities for education and training are increased 
without incurring additional expense. Utilizing formal critiques and AARs from real- 
world incidents is an efficient means for sustaining specialized training and education to 
prepare for responding to homeland security incidents. 

The process values analysis supports the notion that the proposed policy provides 
greater opportunity than the current policy for stakeholder influence on individual and 
organization learning, as well as, surprise avoidance. The proposed policy capitalizes on 
the principles of andragogy to encourage learners to take responsibility for their learning 
through active engagement. The format used to conduct formal critiques promotes 
investigating topic areas that are of particular interest to the actual learners, as opposed to 
the content focus being selected by a small number of management personnel like is seen 
with the current policy. Using AARs as training tools creates opportunities for vicarious 
learning both immediately and in the future. Firefighters can choose to independently use 
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these documents at any point to further their personal bank of knowledge. Formal 
critiques and AARs provide multiple opportunities for stakeholders to engage in and 
influence the learning process. 

Analysis of robustness and improvability reveals that the proposed policy is a 
more viable option in terms of budget, time requirements for implementation, 
administrative complexities, and implementation flexibility. Consistent with the findings 
from the efficiency analysis, formal critiques and AARs are a more economical option 
than traditional classroom-based training and HSEEP-compliant exercises. The time 
required for implementation is also less, with the shorter duration between the decision to 
hold an event and event execution improving execution abilities. Participant scheduling 
to attend each of the training option presents some challenges, but the current policy 
requires that special procedures be implemented while the proposed policy utili z es 
procedures used by the organization on a daily basis. Einally, the proposed policy is 
more advantageous in terms of flexibility than the existing policy since those in charge of 
execution have more freedom to customize the delivery in order to more fully address the 
individual needs of the group. 

In conclusion, the analysis contends that the proposed policy provides greater 
benefit for the investment of resources than the current policy option. The proposed 
policy is more efficient in terms of financial cost and time investments. It also creates 
greater opportunities for stakeholders to influence individual and organizational learning, 
as well as encourages firefighters to assume a greater responsibility for their own 
learning. Eastly, implementation of the proposed policy is more robust than the current 
one and provides the ability for customization to meet the specific learning needs of 
stakeholders. 
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VI. RECOMMENDATIONS 


The extent to which formal critiques and AARs can be used as training tools 
throughout an organization is only limited by the user group’s level of creativity; 
providing a tool that empowers firefighters to build upon their existing bank of 
knowledge to enhance individual, fire company, and organizational capabilities for 
responding to incidents of asymmetric origin. The incorporation of the use of formal 
critiques and AARs from the response to conventional incidents into existing homeland 
security training and education strategies creates a dynamic opportunity for sustaining 
existing capabilities while building new ones. Integration of these smart practices 
(Bardach, 2009) into existing training and education programs helps to create a 
decentralized system for sustaining homeland security training and education that utili z es 
both instructor-led and learner-focused opportunities (Brafman & Beckstrom, 2006). A 
decentralized training approach is likely better able to withstand the ever-present 
challenges encountered in administering training and education programs within the fire 
service because the diverse tactics used in a mixed-methods approach are dependent on 
different enabling factors, such as time, money, and personnel resources. The inability to 
implement one tactic does not preclude the execution of another. 

The analysis of the two target capabilities provides support for this thesis’ claim 
that firefighters use comparable skill sets when responding to certain types of 
emergencies that may originate from both conventional and asymmetric causes. 
However, a single conventionally caused incident of typical scope and nature alone is 
likely of smaller scale than a homeland security incident and does not provide the 
opportunity to execute all activities and tasks for the designated capability. This finding 
supports the recommendation to incorporate formal critiques and AARs into daily 
operations instead of using them sporadically. 

Expansion of the scope of formal critiques and AARs to include “what if’ 
questions about variables that may occur during incidents of similar nature but of 
different origin creates a value innovation (Kim & Mauborgne, 2005) for homeland 

security training and education (see Figure 1). Value innovation is a business strategy 
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term used by Kim and Mauborgne (2005, p. 16) to describe actions that favorably affect 
both cost structure and the value to the customer. Cost saving are made by eliminating 
and reducing factors that the business competes on. Customer value is increased by 
raising and creating elements that the industry has never offered. 

Formal critiques and AARs from conventional emergencies used as training and 
education tools to help firefighters prepare for responding to asymmetric incidents creates 
a new value proposition (Johnson, 2010) for sustaining homeland security training and 
education. A value proposition is a term used in business models that identifies a job that 
needs to be done and evaluates the possible options for accomplishing the job in a 
manner that is efficient, reliable, and reproducible. In this case, the job to be done is the 
provision of homeland security training and education to firefighters. The possible 
options for meeting the learning needs are classroom training, HSEEP-compliant 
exercises, and expanded formal critiques and AARs that include “what if’ questions 
about variables that may occur during similar incidents that originate from different 
origins. The findings in this thesis project suggest that formal critiques and AARs 
embody the three criteria for a value proposition in a way that is more advantageous to 
fire departments for sustaining specialized training and education than the more 
traditional options for training and education. 

A strategy canvas (Kim & Mauborgne, 2005) is another tool used in business 
models to pictorially graph the differences among possible options in terms of each 
option’s the level of offering for the primary factor under consideration. The main 
component of a strategy canvas is the value curve, or the graphic depiction of each 
possible option in relation to the other options. As seen in Eigure 1, using expanded 
formal critiques and AARs from conventional emergencies as tools for preparing 
firefighters to respond to incidents with asymmetric origin redefines the existing 
homeland security training and education value curve. 
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Figure 1. Homeland Security Training and Education Strategy Canvas 

A. VALUE CURVE REDEFINITION 

Value curves are redefined by completing four actions: eliminating factors that 
the industry takes for granted; reducing factors well below the industry standard; raising 
existing factors above the industry standard; and creating factors that the industry has 
never offered (Kim & Mauborgne, 2005) (see Figure 2). Expanded formal critiques and 
AARs accomplish these four actions by: eliminating the dependence of grant funding to 
create learning opportunities; reducing resource requirements and the probability of 
predictable surprise occurrence; raising the number of individual and organizational 
learning opportunities and the number of firefighters impacted by each learning 
opportunity; and creating a sustainable homeland security training and education strategy 
that promotes an andragogy-based learning environment and opportunities for front-line 
personnel to participate in policy development. 
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Figure 2. Eliminate-Reduce-Raise-Create Grid 

1. Eliminate 

Formal critiques and AARs from conventional emergencies eliminate the 
dependence on dedicated funding sources for the creation of learning opportunities. 
Firefighter salaries and equipment costs are expenses included in a department’s annual 
budget. These expenses remain irrespective of the types of on-duty activities in which 
firefighters engage. A change in the level of Homeland Security Grant Program funding, 
the funding source that is commonly relied upon by fire departments to support homeland 
security activities, does not negatively impact abilities to conduct formal critiques and 
write AARs. 

2. Reduce 

Expanded formal critiques and AARs reduce the human and financial resources 
invested in creating learning opportunities that help firefighters prepare for responding to 
incidents of asymmetric origin. The expenses of instructor fees and participant 
backfill/overtime are reduced since the frequency of need for standalone classroom-based 
training is decreased. Similarly, personnel and equipment costs associated with HSEEP- 
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compliant exercises are reduced since the need to conduct exercises to practice activities 
and tasks that are routinely executed in the course of conventional fire service activities is 
decreased. The new strategy does still require an investment of personnel time to prepare 
for a critique and write the AAR, but this investment is less than the typical amount of 
time invested in planning and coordinating department-wide training initiatives and 
HSEEP-compliant exercises. 

Eormal critiques and AARs position frontline personnel and administration to 
learn collaboratively from past experiences in order to prepare for future situations. The 
expanded critique process creates opportunities to identify the multiple factors that may 
contribute to surprise and discuss the procedures and tactical decisions that can reduce 
the possibility of encountering surprise during incident mitigation activities. Bayesian 
theory postulates that surprise can only occur in the presence of uncertainty (Itti & Baldi, 
2005). Routine engagement in individual and organizational learning reduces uncertainty 
of abilities to make decisions and execute activities and tasks during situations of 
complex nature, ultimately reducing the probability if experiencing a predictable surprise. 

3. Raise 

Expanded formal critiques and AARs raise the number of opportunities for 
individual and organizational learning. The resource investments required for the 
delivery of a DHS-approved training curriculum and the conduct of an HSEEP-compliant 
exercise limit the number of events that can be supported per year, in turn, limiting the 
frequency and scope of opportunities to learn. Conversely, formal critiques and AARs 
provide an occasion to learn each time an incident requires the use of some of the 
activities and tasks contained on the TCE. The breadth of incident types, activities, and 
tasks covered during a formal critique is dictated by a fire department’s capabilities and 
the types of incidents to which fire companies are routinely dispatched. This variety is 
likely greater than the focused content of pre-planned events, enabling each AAR to 
contain multiple learning points. 

The number of firefighters impacted by each learning opportunity is also raised. 
The ability to conduct formal critiques and write AARs is dictated by the unpredictable 
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occurrence of specific types of emergencies. However, it is this author’s experience that 
completing a minimum of six formal critiques and AARs in a 12-month time period is a 
reasonable expectation. Distribution of AARs to the entire workforce raises the number 
of firefighters impacted by each learning opportunity. 

4. Create 

Expanded formal critiques and AARs create a sustainable homeland security 
training and education strategy that is based on adult learning theory. In light of the ever¬ 
present challenge to do more with fewer resources, exploiting the use of formal critiques 
and AARs from conventional incidents provides an opportunity to leverage resources in a 
manner that supports the simultaneous accomplishment of multiple objectives. For 
example, the financial savings realized by exploiting the use of formal critiques and 
AARs instead of conducting staged exercises that duplicate practice of the same activities 
and tasks can be used to pay for firefighters to attend technical training. The analysis 
from this project suggests that between nine and 13 firefighters can attend technical 
training for each formal critique that is used in place of a staged exercise29’30 xhe 
money saved can be invested in providing training to additional personnel, creating a 
mechanism for a fire department to sustain costly special operations training programs, 
without compromising overall learning goal and objectives. Furthermore, creating a 
resource bank of archived AARs facilitates learning at a later date by creating content for 
company schools and independent study. 

Formal critiques create opportunities for front-line personnel to participate in 
policy development. Collaboration among frontline and supervisory personnel increases 
the perspectives from which an issue is considered. The more comprehensive the 
dissection of cause and effect relationships, the greater the probability that a fire 
department will identify which policies and procedures should be memoriali z ed and those 
that need to be changed. Collaborative brainstorming helps to create options for 

29 For each hazardous materials HSEEP-compliant exercise not conducted, the cost savings can be 
used to send nine—12 firefighters to hazardous materials technical training. 

30 For each search and rescue HSEEP-compliant exercise that is eliminated, the money saved can be 
used to send nine—13 firefighters to for technical training. 
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strengthening identified deficiencies that are realistic in terms of implementation and 
adequacy for correcting the problem. Participation in policy development improves 
stakeholder buy-in, which helps to create an organizational culture that is conducive to 
enacting policy changes. 

The use of these smart practices enables administrators to meet daily operational 
demands, while creating learning opportunities for effectively responding to incidents of 
conventional and asymmetric origin. Additionally, opportunities are created to build the 
collaborative capacity necessary to critically assess and improve department policy in a 
comprehensive manner. 

B. ACTION STEPS 

The desired value proposition for sustaining homeland security training and 
education can be achieved by expanding the scope of formal critiques and AARs from 
conventional incidents to include “what if’ questions about incidents of similar nature 
with asymmetric origins. To successfully assimilate this strategy into routine operations, 
it is recommended that a fire department execute four activities: 

• Commit to learning from crisis; 

• Re-evaluate existing homeland security training and education programs; 

• Establish policy for conducting formal critiques and writing AARs with an 
expanded scope; and 

• Periodically evaluate response activities in terms of the TCL. 

The findings from this thesis project suggest that formal critiques and AARs as 
mechanisms for learning have not been fully institutionalized and that valuable learning 
opportunities are possibly being missed (Espedal, 2008; Holmqvist, 2009; Carmeli & 
Sheaffer, 2008). These findings are consistent with Smith and Elliott’s (2007) assertion 
that organizations are resistant to learning from crisis. To effectively incorporate the use 
formal critiques and AARs from conventionally caused incidents as tools to sustain 
homeland security training and education, an organization must commit to not only 
learning for crisis, but to also learning from crisis. An environment that encourages 

constructive criticism from all personnel ranks must be created, with department 
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leadership willing to make policy modifications as necessary to support the findings from 
careful analysis of identified areas for improvement. A review of disciplinary procedures 
for job performance deficiencies may be necessary to promote a critique environment that 
is conducive to honest identification of cause and effect relationships. Cultivation of 
participant confidence that formal critique participation is purely for the purposes of 
individual and organizational learning without punitive ramifications improves the 
quality of the discussion and learning. 

Consistent dedication of the resources necessary to conduct formal critiques and 
write AARs is necessary, even when this means adjusting the priority ranking of non¬ 
emergency organizational activities to accommodate the unpredictable schedule in which 
incidents that employ the activities and tasks contained in the TCL occur. The transition 
of moving from a culture that is focused on learning for crisis to one that includes 
learning from crisis requires a commitment of time, energy, and resources to overcome 
the challenges confronted when moving through the process of change. However, 
organizations that persevere in proactively working to learn from their own experiences 
will likely decrease the probability of experiencing predictable surprises and lessen the 
likelihood of firefighter death or injury from a cause that is retrospectively judged as 
preventable. 

A department’s homeland security training and education program should be re¬ 
evaluated, with the intent of identifying opportunities to adopt new strategies for 
achieving the defined goals and objectives. The opportunity for fire departments to 
lessen the possibility of predictable surprises during homeland security incidents exists, 
but requires policy-makers to incorporate different methods of learning into existing 
organizational practices (Smith & Elliott, 2007; Carmeli & Sheaffer, 2008; Holmqvist, 
2009). Cross-walking the TCL with the agency’s core mission, areas of responsibility, 
and emergency incident trends identifies potential opportunities for using expanded 
formal critiques and AARs from conventional incidents to support specialized training 
and education goals and objectives. Comparison of the newly created list with the 
expected outcomes when one of the traditional training and education tools are employed 
creates an awareness of possibilities for using new strategies to supplement the more 
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conventional methods. Adult learning theory principles, vicarious learning opportunities 
created, execution frequency, and the breadth and variety of content are factors to 
consider when selecting tools for meeting learning objectives. Similar to any policy 
decision, a compromise on some factors may be necessary. Policy-makers must make 
choices that best achieve the overall goals with respect to the available resources. 

Policy creation clarifies the expectations for how formal critiques and AARs from 
conventional incidents are used to sustain homeland security training and education, as 
well as the roles and responsibilities of department members for policy execution. 
Specification of the procedures for policy execution creates implementation uniformity, 
while helping to ensure that the end product is a useful learning tool. For example, 
creating a formal critique facilitator guide provides individuals responsible for 
conducting critiques, both those who are new to the facilitation process and those who are 
experienced educators, with information on how to conduct the meeting in order to 
optimize participant and organizational learning. Similarly, developing an AAR 
document template that details the expected contents of each section helps to ensure that 
information is written and organized in a manner that promotes independent learning. A 
simple narrative of the incident with an analysis of actions, recommendations on how to 
strengthen capabilities, and pictures to provide context for the AAR content may be all 
that is required in an AAR to facilitate vicarious learning. This author has experimented 
with developing AARs for conventionally caused emergencies in Chicago and has found 
such a document structure to be an effective format for accomplishing learning 
objectives. See the Appendix for an example. Policy and procedure development for 
formal critiques and AARs as a means to sustain homeland security training will likely 
involve a process of trial and error. Uniform implementation of each documented policy 
version enables administrators to effectively assess which aspects of a policy are effective 
and those that require further refinement. 

Periodic evaluation of response procedures in terms of the TCL is necessary to 
ensure the on-going relevance of the findings from this thesis project to a fire 
department’s homeland security training and education goals and objectives. The new 
policy option is based on the finding that firefighters execute the same activities and tasks 
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for similar incident types, irrespective of whether the incident originated from a 
conventional or asymmetric cause. Therefore, assessing the continued accuracy of this 
assertion is needed in order to remain confident in the new strategy for training and 
education. Routine re-evaluation is recommended, but should minimally be done when 
the threats faced by and response capabilities of a fire department change and when there 
are changes made to federal capability recommendations. Duplication of the process 
used in this thesis project to assess the similarities and differences between the response 
to conventional and asymmetrically caused emergencies is likely sufficient to validate 
continued applicability of the new learning strategy. However, fire department 
management may wish to consider real-time evaluation of response procedures in terms 
of the activities and tasks for each target capability since a real-time evaluation by on¬ 
scene chief officers will likely be less time consuming than a retrospective analysis. The 
HSEEP EEGs can still be used for a real-time evaluation. 

In summary, incorporating formal critiques and AARs from conventional 
incidents into homeland security training and education programs helps to decentralize a 
fire department’s strategy for learning and improves the sustainability of specialized 
programs. The new value curve reduces resource requirements and the probability of 
predictable surprise. The dependence on external resources for creating learning 
opportunities is eliminated. At the same time, the redefined value curve raises the 
number of opportunities for learning and the number of firefighters impacted by each 
opportunity. A sustainable training and education program that is sensitive to the 
principles of adult learning theory and provides opportunities for all stakeholder levels to 
engage in the policy development process is created. 

To achieve the new value curve, engagement in four activities is recommended: 
committing to the process of learning from crisis, re-evaluating current homeland security 
training and education programs, establishing policy to support implementation of this 
new strategy, and routinely evaluating emergency response requirements in terms of the 
TCE. Execution of these recommendations establishes an organizational culture in which 
formal critiques and AARs from conventional incidents can be used successfully to 
sustain homeland security training and education programs. 
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VII. CONCLUSION 


Formal critiques and AARs from conventional emergencies are tools that can be 
used to sustain homeland security training and education. The numerous similarities 
between the activities and tasks performed to mitigate emergencies of similar type, 
irrespective of whether the incident originated from a conventional or asymmetric cause, 
creates opportunities for firefighters and their organization to use real-world emergencies 
to prepare for the homeland security challenges they may one day face. The cost-benefit 
analysis conducted in this thesis project suggests that incorporating the use of formal 
critiques and AARs into homeland security training and education programs is an 
efficient use of resources when compared with the resource requirements for conducting 
stand-alone, classroom based training and Homeland Security Exercise and Evaluation 
(HSEEP)-compliant exercises. Eurthermore, the analysis indicates that the proposed 
policy option is more advantageous than traditional training and education methods 
commonly used to strengthen homeland security-related capabilities when process values, 
robustness, and improvability are factors considered by policy makers. 

Integration of these smart practices into routine department operations increases 
the number of opportunities created for firefighters to learn, from both their personnel 
experiences and vicariously through the experiences of colleagues. The phrase “practice 
makes perfect” is commonly used to describe the value of routine performance of 
essential skills. However, the phrase “practice makes permanent” has greater 
applicability when working to decrease the probability of predictable surprises through 
training and education. The more frequent the opportunities that firefighters have to 
practice critical activities and tasks, the greater the probability of successful performance 
when called upon to execute these procedures to mitigate emergencies of diverse origin 
and complex nature. Very few emergencies occur according to a textbook script. 
Eirefighters must be able to adapt to the situation encountered and safely execute the 
activities and tasks required to protect the health, safety, and property of those they serve. 
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A. LIMITATIONS 

A primary limitation of this thesis project is the sample size and the fact that the 
entire sample is taken from the same data source. This project uses four cases to analyze 
two target capabilities. The cases selected are subjectively assessed as representative of 
the types of incidents to which the CFD responds. However, the small sample size 
allows one to question whether or not there is something unique about these cases that 
contributes to the findings that would not be found in the documentation from other 
incidents of similar type. While using data from the same source creates consistency 
among data elements (e.g., salary costs, fire company responsibilities), it does not 
account for differences among urban fire departments or assess how these differences 
may impact the findings. With Chicago being the third largest city in the United States 
and uniqueness of geography, demographics, and critical infrastructures, it is possible 
that the incidents to which CFD routinely responds may be of unique complexity when 
compared with other urban fire departments. 

This thesis project relies entirely on historical documentation. No real-time or 
qualitative data is used. Real-time and qualitative data may provide a greater 
understanding of decision-making processes for why an activity or task was not 
performed or if it was performed and just not documented. 

Finally, there is no control for variability among fire company assignments and 
the experience levels and documentation styles of each company officer and chief. 
Documentation for each case is aggregated, but the documentation style of each officer 
and chief varied. It is unknown if this is a result of experience level, personal style, the 
assignment that each company was given to perform, and/or a combination of multiple 
factors. 

B. AREAS FOR FURTHER RESEARCH 

To validate the findings from this thesis project, application of the research 
methodology to cases obtained from a different urban fire department is warranted to 
determine if the findings are replicable. Future researchers should consider including 
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real-time evaluation of emergency response operations and qualitative interviews in the 
research methodology to further enhance the quality of the information used for analysis. 
An increase in the sample size and including data from more than one fire department 
will strengthen confidence in the reliability of findings. 

The TCL has 37 capabilities, of which 21 are solely focused on the response 
mission area. Future researchers may wish to analyze whether or not the activities and 
tasks for additional capabilities are routinely executed when fire departments respond to 
emergencies of conventional origin. If such findings are consistent with those of this 
thesis project, the proposed policy option can be expanded to include additional 
capabilities. 

The response to a homeland security incident requires the involvement of multiple 
agencies and disciplines, or a megacommunity (Gerencser, Lee, Napolitano, & Kelly, 
2008). Some of the individual target capabilities contain activities and tasks that reflect 
this megacommunity involvement. Future researchers may consider analyzing the 
comprehensive list of target capabilities that are executed by the megacommunity in 
response to a conventional incident, with the intent of expanding the proposed policy 
option beyond use by one agency. Incorporating the megacommunity into the formal 
critique and after action process may provide valuable opportunities for the entire group 
to prepare for homeland security incidents by optimizing learning opportunities from 
conventionally-caused incidents. 
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APPENDIX: SAMPLE ALTER ACTION REPORT 


Trench Rescue 
After Action Review 

Incident Date: May 27, 2009 
Incident Location: 5900 W. Belmont Ave 
After Action Date: June 2, 2009 

Incident Overview 

At 1225, a member of a construction company called 911 to report that one of their 
employees was buried up to his knees in a below grade hole secondary to the earth caving 
in. Upon arrival at the scene, Engine 94 and the Battalion Chief confirmed that one 
person was trapped in an unsupported trench and that the trench was approximately five 
feet by five feet and ten feet deep. The victim reported that his legs were pinned 
underneath a pipe and that they hurt. The Chicago Fire Department (CFD) initiated a 
trench rescue response. Thirty-one companies responded to the scene and mitigation of 
the incident took approximately five hours. 

Analysis of Actions 

1. The first on-scene companies appropriately used the plywood that was available 
at the construction site to brace the hole, provided the victim with oxygen, tied a 
rope around the victim’s body, gave the victim a shovel, and called for a full 
trench rescue response. The companies did not try to pull the victim from the 
hole. The companies followed departmental procedure and correctly 
implemented basic bracing techniques learned through training. 

2. After using plywood to further brace the hole, squad company members assessed 
the situation further and learned that the most unstable area was the concrete on 
the north side of the hole, that a drain tile had broken and washed pea gravel into 
the trench, that there was a catch basin in the hole, and that the victim’s legs were 
in fact bent underneath him and caught in a pipe. Members placed struts to 
further stabilize the hole, with a priority of protecting the victims head and chest. 
The air bags and plywood placed were successful in stopping the influx of pea 
gravel. Operations were stopped at frequent intervals so that supervisory 
personnel could ensure that the shoring was correct and that the appropriate safety 
measures were in place. Members followed established departmental protocols 
for trench operations. 

3. Once the hole was stabilized, squad members rotated going into the hole to dig 
the debris away from the victim. At all times, two members were harnessed and 
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in position to initiate secondary rescue operations, if needed. Members followed 
established departmental safety standards for trench operations. 

4. Team companies set-up the cutting station, brought struts and the paratech system 
for inflating the air bags to the hole, kept the area of the collapse clear of 
pedestrian traffic, and provided generalized incident support. The precision of 
measurements and wood cutting greatly decreased the time required for incident 
mitigation. 

5. En-route, the Fire Alarm Office provided responding companies with a status 
update via the Mobile Data Computer. Receipt of this information prior to arrival 
on-scene helped responding companies pre-plan and prioritize the actions they 
would taken upon arrival. 

6. EMS personnel started an IV and administered IV fluids, pain medications, and 
other medications as appropriate for a victims with crush injuries. Paramedics 
routinely monitored the victim’s blood sugar and calculated rates of fluid 
replacement. EMS considered requesting a helicopter for transport; however, the 
amount of debris on-scene would have created safety hazards. EMS provided 
excellent care to maintain patient stabilization through the entire incident and 
subsequent transport. 

7. The Water Department’s vacuum truck was brought to the scene to help remove 
the earth faster. The truck’s hose is 8” in diameter. Due to the size of the hole 
and the on-going caving of the surrounding earth, a smaller diameter nozzle was 
needed to better control the rate of removal. CFD and Water Department 
personnel worked together to create a makeshift nozzle of smaller diameter out of 
duct tape, a five-gallon bucket, and an orange traffic cone. The new nozzle 
worked to control the sucking and facilitate removal of the earth trapping the 
victim. CFD has since received the smaller diameter nozzles that were ordered 
prior to the incident. 

8. Fire Ground was used for on-scene communications; however, a runner was used 
for relaying requests between the hole and the cutting station. A dedicated 
channel for communication between members operating at the hole and the 
cutting station is included in the general order on trench rescue operations that is 
being written. 

9. While executing primary rescue operations, officers were actively developing 
contingency plans. If primary operations had failed and the victim was in fact 
found to be pinned behind the catch basin, companies had planned to dig through 
the catch basin to perform the extraction. This operation would have been 
complicated by the fact that there was a distance of 20’ between the top and 
bottom of the catch basin. Officers displayed critical thinking in developing 
contingency plans. 
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10. Squad members prepared an A-frame to lift the victim from the hole, however, 
the victim climbed up the ladder himself and the A-frame was not needed. 
Preparation of this device in anticipation of possible need was appropriate. 

11. Tape was put up around the perimeter and lines were painted on the ground to 
identify restricted areas, however, these measures were not independently 
sufficient to restrict access to the scene. 

12. A responder rehab area was established and utilized. Officers did a good job of 
ensuring that members were given sufficient periods for rest and rehab. 

Recommendations/Improvement Plan 

1. Include the vacuum truck in the general order that is being written as a resource to 
be considered requesting. 

2. Continue to include training on use of the vacuum truck during trench rescue 
training classes. Consider joint training with personnel from the Department of 
Water Management on using the truck. 

3. Incorporate round trenches in training classes. Classes typically focus on 
rectangular trenches. Inserting struts to brace a round trench creates unique 
challenges. 

4. Continue to send members of special operations and team companies to trench 
and confined space classes. 

5. Assign personnel to restrict pedestrian traffic to unstable or dangerous areas 
within the scene. Response personnel who are not providing direct service or 
support should not access restricted areas. 

Impact of Training 

AAR participants were asked whether or not participation in special operations training 
classes has had an impact on the way response operations for trench rescue are conducted 
and, if so, to describe how. 

1. A Lieutenant with 23 years of experience said, “Trench operations training has 
helped to minimize the risk to fire fighters. Previously, we jumped right in the 
hole. Now we stabilize the hole first.” 

2. A District Chief with 29 years of experience said, “Training has really helped to 
facilitate companies working together. The impact of cross company training and 
drilling was evident during the entire incident. The technical knowledge of 
responding companies on trench operations was excellent.” 
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3. A company officer said, “Having all officers on the same page, using universal 
language, and having an understanding of what was going on and why really 
helped to facilitate operations.” 

4. An officer of the squad said, “We knew that we had to have a Plan B and Plan C 
incase our initial strategy did not work. We started working on the contingency 
plans right away.” 

5. A firefighter said, “Everyone knew what was being requested. We did not waste 
time trying to figure out what the different pieces of equipment are.” 

6. The Assistant Deputy Fire Commissioner for Special Operations said, “Trench 
operations normally take 8-10 hours. This one took 5 hours. The teamwork and 
efficient manner in which operations were conducted really helped to spare the 
victim further injury.” 

7. A squad officer said, “Skills learned during collapse, trench, rope, and confined 
space classes were used in this response.” 

8. An investigator from the Occupational Safety and Health Administration (OSHA) 
complimented CFD on adhering to safety standards, especially wearing harnesses 
when working around the hole. 

Summary 

In many ways, this trench rescue operation was a success. The victim was transported to 
the hospital in stable condition and discharged from the hospital the next day. 
Throughout the rescue operation, teamwork among Fire Department personnel prevailed 
and Departmental safety standards were maintained. This incident demonstrates the 
value of the financial investment that has been made in training personnel to perform 
rescue operations in a safe and effective manner. The victim was successfully rescued 
and there were no injuries to any on-scene rescue personnel. 
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